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ABSTRACT 



A computer system for controlling a screen display state of 
a monitor according to input of a password for protecting 
confidential information from unauthorized users when a 
computer system is between a standby mode and a suspend 
mode. The computer system comprises a power controller 
for converting an operation mode of a computer system into 
a power management mode which represents one of a 
power-on mode, a standby mode, a graphic off mode, and a 
suspend mode according to an access state of the computer 
system; and a screen controller for converting a screen 
display state of the display according to a designated one of 
the power management mode, and controlling the screen 
display state of the monitor according to input of a password 
if the computer system is accessed in the graphic off mode. 

13 Claims, 5 Drawing Sheets 
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COMPUTER SYSTEM AND METHOD FOR Provision Of Power To Computer Peripherals issued to 

CONTROLLING SCREEN DISPLAY OF A Tornai, U.S. Pat. No. 5,410,713 for Power Management 

MONITOR IN A POWER MANAGEMENT System For A Computer issued to White, and recently U.S. 

MODE Pat. No. 5,483,464 for Power Saving Apparatus For Use In 

5 Peripheral Equipment Of A Computer issued to Song and 

CLAIM FOR PRIORITY assigned to the same assignee of the present invention. 

™ . .. . , . , Usually, the monitor is shut down during the period of 

This application makes reference to, incorporates the ^activity with the exception of a small amount of power 

same herein, and claims all benefits accruing under 35 nece ssary to detect when the computer system becomes 

U.S.C. §119 from an application for COMPUTER SYSTEM active again as t0 resU me operation of the monitor. 

BEING CAPABLE OF CONTROLLING SCREEN DIS- 10 Generally, when the monitor is powered normally, the dis- 

PLAY STATE AND METHOD THEREOF earlier filed in p i ay of data image is blanked during the period of inactivity 

the Korean Industrial Property Office on the of Jun. 16 th and re-displayed when the computer system becomes active, 

1997, and there duly assigned Ser. No. 24852/1997, a copy i.e., when an input device such as a keyboard is operated, 

of which application is annexed hereto. During this type of blanking, however, the monitor contin- 

15 ues to consume normal power. 

BACKGROUND OF TOE INVENTION Display powef management standards have been M to 

1. Technical Field save power consumption by controlling monitor power with 
The present invention relates to a computer system for r fP ect t0 the °P e «ti°°al status of the computer system. In 

controlling a screen display of a monitor in a power man- , n the P°wer management mode power supply * managed 

t , j -.-11 1** , 20 according to the operational status of the computer system, 

age ment mode, and more particularly, relates to a computer j r . . * • 1 •« j • * 

to r «• . . 1 r . r . I ne mode status 01 the power management is classified into 

system lor controlling a screen display 01 a monitor in a . « j 1 <r ■-■ * u *u 

J « . . . £ . power-on, standby, suspend and power-off. First, when the 

power management mode according to input of a password * . • c * * j • *■ 1 

K . 1 • c r • ■ « computer system is first turned on, power is continuously 

tor protecting confidential information from unauthonzed v j * u j * r*u * * • 

1 *l . . 1 j r supplied to each device of the computer system in a power- 
users when the computer system is reawakened from a j aa • j c • *• ** *u . * 

ra hie off mode mode. Alter a period of inactivity, the computer system 

S P • is switched to a standby mode to reduce power consumption 

2. Related Art by lowering the operation frequency of a central processing 
Contemporary computer systems generally consist of a ua it (CPU), turning off operation of a monitor, and turning 

main unit (which may have built-in storage devices such as 0 ff operation of a hard disk drive (HDD). The computer 

floppy disks, hard disks and CD-ROM), a keyboard and a 30 system may also be temporarily suspended in a suspend 

monitor. The main unit may be connected to a cathode -ray mo de when there is a sudden power failure or when the 

tube (CRT) monitor and other peripheral devices. In normal computer system is not accessed for a predetermined time 

usage, it is common for the monitor and other peripherals to period. Lastly, the computer system may be turned off 

be turned on and to remain on for as long as the computer completely in a power-off mode. 

is running, even though the peripherals are actually used 35 During the standby mode, the computer system can be 
only a small percentage of the time. A typical color video automatically awakened and the mode of the computer 
monitor, for example, may consume as such as 50 to 80 sys t em is converted back into the full power-on mode 
percent of the total electrical energy consumed by a personal without checking a password when there is an external input 
computer (PC). If the monitor consumes valuable energy such as input from a keyboard, a mouse, and an infrared port, 
only to remain idle, not only the valuable energy resources 40 and activity from a hard disk drive (HDD), a floppy disk 
will be wasted but the fife of the monitor will decrease dr j ve (FDD), a fax/modem card, and a network card. During 
rapidly. Obviously, power can be conserved if the user turns the suspend mo de, the computer system provides a visual 
the computer system off or in a standby mode each time the display of identification information such as "enter pass- 
computer system is no longer in use for a prolonged period. word » on a mon itor when the power switch is turned on if 
This requirement is, however, impractical. Therefore, sys- 45 the password is set at the time of the computer setup, and the 
tern should be designed to automatically conserve valuable mode 0 f the computer system is converted into the full 
energy resources when the computer system is not in use. power-on mode after checking that the input password is 
In principle, automatically saving energy by turning off correct. The mode of the computer system is converted into 
the computer system is fairly easy. Computer systems are the full power-on mode without checking the password if the 
invariably designed so that if there is no activity from the 50 password is not set at the time of the computer setup, 
keyboard or from other external inputs during a specified xte mode of the computer system is converted into the 
time, the computer systems are turned off or placed in a ^\\ poW er-on mode after checking the password if the 
standby mode to save energy. Only minimal logic is kept password is set at the time of the computer setup when the 
active to detect when the keyboard or other external inputs power swilCD ^ mraed orlj and the mode of the computer 
are becoming active again so as to turn the computer system 55 sys te m is converted back into the full power-on mode 
back on. Conventional power saving features for typical without checking the password if the password is not set at 
personal computers having monitors as separate items are the time 0 f the computer setup. However, typical computer 
disclosed, for example, in U.S. Pat. No. 5,163,124 for system, as I have observed, has a disadvantage in that 
Method And Apparatus For Controlling Power To Device In information can not be kept confidential from unauthorized 
A Computer System issued to Yabe et al., U.S. Pat. No. 60 users smce tne password is not needed, before the computer 
5,218,704 for Real Time Power Conservation For Portable system is conve rted into a suspend mode, to convert the 
Computers issued to Watts, Jr., U.S. Pat. No. 5,375,245 for computer system back into a full power-on mode. 
Apparatus For Automatically Reducing The Power Con- 
sumption Of A CRT Computer Monitor issued to Solhjell et SUMMARY OF THE INVENTION 
al., U.S. Pat. No. 5,389,952 for Low-Power-Consumption 65 Accordingly, it is therefore an object of the present 
Monitor Standby System issued to Kikinis, U.S. Pat. No. invention to provide a computer system for controlling a 
5,408,668 for Method And Apparatus For Controlling The screen display of a monitor in a power management mode. 
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It is also an object to provide a computer system for DETAILED DESCRIPTION OF THE 

protecting confidential information from unauthorized users PREFERRED EMBODIMENT 

when a monitor is reawaken from a standby mode. Referring now to the drawings and particularly to FIG. 1, 

It is another object to provide a computer system for which illustrates a computer system for controlling a screen 

operating in a graphic off mode to turn off a screen display 5 display of a monitor in a power management mode. As 

of a monitor for protection against unauthorized users after shown in FIG. 1, the computer system includes a system bus 

the monitor is in a standby mode for a predetermined time 1> a central processing unit (CPU) 2, a display device 3 such 

period as a CRT monitor, a chip set 4, an input/output controller 7, 

It is further an object to provide a computer system for an ^vice electronics (IDE) controller 8, an 

t e J . j • • | *\. 10 expansion function block 9, a random-access-memory 
requesting input of a correct password while operating in a 51 a readK)mv _ memory (ROM) 52, a keyboard 71, 
graphic off mode before returning to a mil power-on mode. a mouse ^ a floppy ^ ^ ( VdD) 73, an infrared port 
These and other objects of the present invention can be 74, a hard disk drive (HDD) 81, a compact disk read-only- 
achieved by a computer system having a central processing memory (CD-ROM) 82, an audio card 91, a fax/modem card 
unit, a memory, a system bus and a monitor and comprises 92, and a network card 93. 

a power controller for converting an operation mode of a The system bus 1, is an input/output interface applied to 

computer system into a power management mode represent- a micro-processor, for connecting data, command, and inter- 

ing one of a power-on mode, a standby mode, a graphic-off rupt in the computer system to each circuit block or each 

mode, and a suspend mode according to an access state of device. The CPU 2 is connected to the system bus 1 for 

the computer system; and a screen controller for converting 2Q controlling the computer system. The monitor 3 provides 

a screen display state of a monitor according to a designated visual display of information data on a screen. The chip set 

one of the specific power management mode, and for 4 is connected to the system bus 1 for converting the mode 

controlling the screen display state of the monitor according of the computer system into a specific power management 

to input of a password when the computer system is accessed mode by controlling video signals output from the monitor 

in the graphic off mode. 2g 3, the operation frequency of the CPU 2 and the operation 

According to another aspect of the present invention, a state of tne HDD 81 according to the access state to the 

method for controlling a screen display state of a computer computer system. The RAM 51 is connected to the chip set 

system, comprises the steps of: converting the mode of a 4 and serves as a memory area where the CPU 2 stores a 

computer system into a standby mode if power is applied to system software and a user software, 

the computer system and there is no access to the computer 30 The input/output controller 7 is connected to the system 

system; converting a screen display state of a monitor if bus 1 for controlling the input or output from the keyboard 

there is no access to the computer system for a predeter- 71, the mouse 72, the infrared port 74 and the FDD 73. The 

mined time in a standby mode, and converting the mode of IDE controller 8 is connected to the HDD 81 and the 

the computer system" to a graphic off mode in which the CD-ROM 82 via the system bus 1 for controlling the input 

screen output state of the monitor is controlled according to 35 or output of the HDD 81 and the CD-ROM 82. The ROM 52 

input of a password if the computer system is again is connected to the system bus 1 and stores specific appli- 

accessed; and controlling the power supply provided to the cation programs that are used by the CPU 2. The expansion 

computer system according to the operation of a power function block 9 supports the audio card 91, the fax/modem 

switch by converting the mode of the computer system into card 92 and the network card 93 for function expansions in 

the suspend mode if the computer system is not accessed for 40 the computer system. 

the predetermined lime in the graphic off mode. The monitor 3 includes a graphic memory 31 for storing 

The present invention is more specifically described in the data t0 be displayed on the screen; a graphic controller 32 for 

following paragraphs by reference to the drawings. attached reading the data from the graphic memory 31 and converting 

only by way of example. tne data into R,G,B signals; a first resistor Rl having one 

45 terminal connected to the graphic controller 32 and the other 

BRIEF DESCRIPTION OF THE DRAWINGS terminal grounded in order to control the brightness of the 

A more complete appreciation of the present invention, R ' G '? ^display unit 33 for receiving the R G,B 

and many of the attendant advantages thereof, will become <^™W fr ™ fff™ controller 32 and providing 

readily apparent as the same becomes better understood by 3 ™* A *? l * y ° f daU informatlon on the screen ' . 

reference to the following detailed description when con- 50 ^0. 2 Jlustrates a P rocess of controlling a screen display 

sidered in conjunction with the accompanying drawings in ° f xh ** ™™? uXq J s J* tem * * P° Wer mana S eQ f nt ™ de * ^ 

which like reference symbols indicate the same or similar * he C ™ 2 performs a POST (power on self test) procedure 

components, wherein: for checkin g whether the hardware operates normally, using 

- . , „ a BIOS boot program when the power switch is turned ON 

FIG. 1 is a block diagram of a typical computer system for J5 and fc lfcd tQ , he computer system at ste 201 and 

controlling a screen display of a monitor in a power man- 203 

agement mode, After ^ posT procedure is completed, the power man- 

FIG. 2 illustrates a flow chart of a process of controlling agement mode is ^ and the compuler system is booted 5 

a screen display of a monitor in a power management mode; an 0 p eratm g syste m. A sentence such as "enter password" is 

FIG. 3 is a block diagram of a computer system for 6Q displayed on a screen of the monitor 3 when the power 

controlling a screen display of a monitor in a power man- switch is turned ON if the password is set at the time of the 

agement mode according to the principles of the present computer setup. The booting operation is completed only 

invention; and when the input password is correct, and the user may use the 

FIGS. 4A and 4B illustrate a flow chart of a process of computer system. However, the user can still use the com- 

controlling a screen display of a monitor in a power man- 65 puter system since the booting operation may be completed 

agement mode according to the principles of the present without checking the password if the password is not set at 

invention. the time of the computer setup at step 205. 
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In this situation, the graphic controller 32 of the monitor 
3 reads data information to be displayed on the screen from 
the graphic memory 31, converts the data information into 
analog R,G,B signals and outputs the R,G,B signals. The 
display unit 33 receives the R,G,B signals and displays the 
letters or the image information on the screen if the user uses 
the computer system through input of the keyboard 71, the 
mouse 72, the infrared port 74, the access of the FDD 73, the 
access of the HDD 81, the access of the compact disk ROM 
82, the access of the fax/modem card 92 or the access of the 
network card 93. Reference current Iref determines the 
brightness of the R,G,B signals from the graphic controller 
32, and the reference current Iref is determined by the first 
resistor Rl. 

After the POST procedure is completed, the power man- 
agement mode converts to the full power-on mode at step 
207. The chip set 4 next converts the mode of the computer 
system into a standby mode by lowering the operation 
frequency of the CPU 2, turning OFF the R,G,B signals 
output from the graphic controller 32, and turning OFF the 
HDD 81 at step 211 if there is no input from any one of the 
keyboard 71, the mouse 72, the infrared port 74 via the 
input/output controller 7, access of the FDD 73 via the 
input/output controller 7, access of the HDD 81 and the 
CD-ROM 82 via the IDE controller 8, access of the fax/ 
modem card 92 and the network card 93 via the expansion 
function block 9 for a predetermined time set for converting 
the full on mode into the standby mode at step 209. 

The chip set 4 converts the mode of the computer system 
into the full power-on mode if there is input of a wake-up 
signal from the input/output controller 7, the IDE controller 
8 and the expansion function block 9 to the CPU 2 from the 
keyboard 71, the input from the mouse 72, the input from the 
infrared port 74, the access of the FDD 73, the access of the 
HDD 81, the access of the CD-ROM 82, the access of the 
fax/modem card 92 or the access of the network card 93 at 
step 213. If there is no input of a wake -up signal from either 
the keyboard 71, the mouse 72, or the infrared port 74, 
access of the FDD 73, access of the HDD 81, access of the 
CD-ROM 82, access of the fax/modem card 92 or access of 
the network card 93 at step 213, the chip set 4 checks the 
time when the computer system operates in the standby 
mode and determines whether a predetermined time has 
passed at step 215. 

When a predetermined time has passed and there is no 
input of a wake-up signal, the mode of the computer system 
is converted into a suspend mode from a standby mode at 
step 217. The chip set 4 senses the state of the power switch 
if the mode of the computer system is converted into the 
suspend mode. The process routine jumps to the step for 
performing the POST procedure if the power switch is 
turned ON, whereby the computer system is booted again. If 
the power switch is not turned ON, the power OFF state is 
maintained at step 219. 

As I have observed however, the typical computer system 
has a disadvantage in that data information can not be kept 
confidential from unauthorized users since a password is not 
needed, before a computer system is converted into a 
suspend mode, to convert a computer system into a full 
power-on mode. 

Turning now to FIG. 3, FIG. 3 which illustrates a com- 
puter system for controlling a screen display of a monitor in 
a power management mode according to the principles of 
the present invention. The computer system includes a 
system bus 1, a central processing unit (CPU) 2, a display 
device 3 such as a CRT monitor, a chip set 4, a timer 6, an 
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input/output controller 7, an integrated device electronics 
(IDE) controller 8, an expansion function block 9, a random- 
access-memory (RAM) 51, a read-only-memory (ROM) 52, 
a keyboard 71, a mouse 72, a floppy disk drive (FDD) 73, 

5 an infrared port 74, a hard disk drive (HDD) 81, a compact 
disk read-only-memory (CD-ROM) 82, an audio card 91, a 
fax/modem card 92, and a network card 93. 

The monitor or display 3 includes a graphic memory 31, 
a graphic controller 32, a first resistor Rl and a display unit 

10 33. The chip set 4 is connected to the system bus 1 for 
converting the mode of a computer system into a power 
management mode according to the operation state of the 
computer system by controlling R,G,B signals from the 
display 3, operation frequency of the CPU 2 and the opera- 

15 tion of the HDD 81. The timer 6 is connected to the 
input/output controller 7 for measuring a standby mode 
operation time if the computer system operates in a standby 
mode for a predetermined time, and outputting a graphic off 
signal. The input/output controller 7 is, in turn, connected to 

20 the system bus 1 for converting the mode of the computer 
system into a graphic off mode by outputting a graphic 
cut-off signal if the graphic off signal is input from the timer 
6, checking a password if there is an input signal from a 
keyboard 71, a mouse 72 or an infrared port 74, and 

25 converting the mode of the computer system into a mode in 
which the user can see a screen of display unit 33; of the 
display or monitor 3 by stopping the output of the graphic 
cut-off signal if the password is correct. The second resistor 
R2 has one terminal connected to a node between the first 

30 resistor Rl in the display 3 and the graphic controller 32, and 
the other terminal connected as the input/output controller 7. 

The system bus 1, the CPU 2, the display 3, the RAM 51, 
the ROM 52, the IDE controller 8 and the expansion 
function block 9 of the present invention have the same 

35 function as the typical computer system as shown in FIG. 1. 
Now, the operation of the computer system for controlling 
a screen display state according to the preferred embodiment 
of the present invention will be described in detail with 

jn reference to FIGS. 3 and 4A-4B here inbe low. 

First, the CPU 2 performs a POST (power on self test) 
procedure for checking whether the hardware operates 
normally, using a BIOS boot program when the power 
switch is turned ON and power is applied to the computer 

45 system at steps 401 and 403, 

After the POST procedure is completed, the power man- 
agement mode is set, and the computer system is booted by 
an operating system. A sentence such as "enter password" is 
displayed on a screen of the monitor 3 when the power 

50 switch is turned ON if the password is set at the time of the 
computer setup. The booting operation is completed only 
when the input password is correct, and the user may use the 
computer system. However, the user can still use the com- 
puter system since the booting operation may be completed 

55 without checking the password if the password is not set at 
the time of the computer setup at step 405. 

Here, the graphic controller 32 in the monitor 3 reads 
information data to be displayed on the screen from the 
graphic memory 31, converts the information data into 

60 analog R,G,B signals and outputs the R,G,B signals to the 
display or display unit 33 if the user uses the computer 
system through the keyboard 71, the mouse 72, the infrared 
port 74, the access of the FDD 73, the access of the HDD 81, 
the access of the CD-ROM 82, the access of the fax/modem 

65 card 92 or the access of the network card 93. The display 
unit 33 receives the R,G,B signals and displays the letters or 
the image information on the screen of display unit 33. 
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Reference current Iref determines the brightness of the is a signal input to the CPU 2 from the input/output 

R,G,B signals from the graphic controller 32, and the controller 7, the IDE controller 8 and the expansion function 

reference current Iref is determined by the first resistor Rl. block 9 by the input by the keyboard 71, input by the mouse 

The greater the first resistor Rl, the lesser the reference 72, input by the infrared port 74, access of the FDD 73, 

current Iref, whereby the screen becomes dark. Generally, 5 access of the HDD 81, access of the compact disk ROM 82, 

the value of the reference current Iref is about 3.05 mA since access of the fax/modem card 92 or access of the network 

voltage value of the R,G,B signals from the graphic con- card 93 at step 419. 

troller 32 is about 1.1V and the first resistor has a resistance However, the computer system maintains the graphic off 

value of 360 Ohm. The power management mode converts mo de in which any letters or image information is not 

to the full power-on mode in which power is normally 10 displayed on the screen of the display unit 33 since the 

supplied to each part of the computer system if the computer R,G,B signals output from the graphic controller 32 are not 

system is used as described at step 407. continually input to the display unit 33 owing to the graphic 

The chip set 4 converts the mode of the computer system cut-off signal output from the input/output controller 7. 

into the standby mode by lowering the operation frequency The user enters the password in the graphic off mode in 

of the CPU 2, turning OFF the R,G,B signals from the 35 which any letters or image information is not displayed on 

graphic controller 32, and turning OFF the HDD 82 at step the screen of the display unit 33 of display or monitor 3. The 

411 if there is no input from any one of the keyboard 71, the input/output controller 7 stops outputting the graphic cut-off 

mouse 72, and the infrared port 74 via the input/output signal and directs the R,G3 signals from the graphic 

controller 7, access of the FDD 73 via the input/output controller 32 to the display unit 33 for a visual display so that 

controller 7, access of the HDD 81 and the CD-ROM 82 via 20 the user may see the screen, and the mode of the computer 

the IDE controller 8, access of the fax/modem card 92 and system is converted into the full power-on mode if the input 

the network card 93 via the expansion function block 9 for password is correct after the input/output controller 7 checks 

a predetermined time set for converting the full power-on the password input by the user in the graphic off mode, 

mode into the standby mode at step 409. However, the input/output controller 7 outputs the graphic 

The R,G,B signals are turned OFT 1 if the chip set 4 sets an 25 cut-off signal continually so that the user can not see the 

R,G,B output registered in the graphic controller 32 to an screen if the correct password is not input, and maintains the 

OFF state, and the R,G,B signals are turned ON if the chip graphic off mode in which the R,G,B signals from the 

set 4 sets the R,G,B output register to an ON state, whereby graphic controller 32 can not be input to the display unit 33. 

the R,G,B signals are output to the display unit 33 for a The chip set 4 converts the mode of the computer system 

visual display. 30 into the mode in which the power to the CPU 2, the display 

The timer 6 starts operating if the mode of the computer 3, the FDD 73, the HDD 81 and the other sub-systems are 

system is converted into the standby mode. The timer 6 turned OFF at step 423 if there is no signal inputted to the 

measures the standby operation time if the computer system CPU 2 from the input/output controller 7, the IDE controller 

operates in the standby mode for a predetermined time, and 35 8 and the expansion function block 9 by the input by the 

outputs the graphic off signal to the input/output controller keyboard 71, input by the mouse input by the infrared port 

7. If there is input of a wake-up signal from the input/output 74, access of the FDD 73, access of the HDD 81, access of 

controller 7, the IDE controller 8 and the expansion function the compact disk ROM 82, access of the fax/modem card 92 

block 9 to the CPU 2 by any one of the keyboard 71, the or access of the network card 93 for a predetermined time set 

mouse 72, the infrared port 74, the FDD 73, the HDD 81, the 4Q for the suspend mode in the graphic mode at step 421. 

CD-ROM 82, the fax/modem card 92 or the network card 93 The chip set 4 senses the state of the power switch if the 

at step 413, the chip set 4 senses the wake-up signal and mode of the computer system converts to the suspend mode, 

converts the mode of the computer system into the full A process routine jumps to the step for performing the POST 

power-on mode. procedure if the power switch is turned ON, and the com- 

However, the input/output controller 7 receives the 45 puter system is booted again. The chip set 4 maintains the 

graphic off signal and outputs the graphic cut-off signal if power OFF state if the power switch is not turned ON at step 

there is no input by the keyboard 71, input by the mouse 72, 425. 

input by the infrared port 74, access of the FDD 73, access The present invention relates to an executive routine when 

of the HDD 81, access of the compact disk ROM 82, access suspend information is stored in an auxiliary memory. When 

of the fax/modem card 92 or access of the network card 93 50 the suspend information is stored in a main memory, steps 

at step 413, and the graphic off signal is outputted from the can be added to determine whether there is access to the 

timer 6 at step 415. CPU for the predetermined time after step 425 and to turn 

The graphic cut-off signal outputted from the input/output OFF the power. In addition, the timer of the present inven- 

controller 7 is applied to the first resistor Rl in the display tion and its equivalent measures the standby operation time 

3. Here, the graphic cut-off signal has the higher voltage than 55 if the computer system operates in the standby mode for a 

the voltage of the R,G,B signals outputted from the graphic predetermined time and outputs the graphic off signal, 

controller 32, and prevents the reference current Iref from As described above, the computer system of the present 

flowing. Accordingly, the R,G,B signals outputted from the invention has an advantage in that information is kept 

graphic controller 32 are not output to the display unit 33, confidential by adding the graphic off mode between the 

whereby the user can not see the screen, that is, the mode of 60 standby mode and the suspend mode when the computer 

the computer system becomes the graphic off mode. The system operates in the standby mode for the predetermined 

second resistor R2 is for protecting the graphic controller 32 time. In addition, information can be secured since a pass- 

from the graphic cut-off signal. word is checked in the graphic off mode in which the letters 

The chip set 4 restores the operation frequency of the CPU or the image information may not be displayed. 

2 to the normal state, turns ON the R,G,B signals output 65 While there have been illustrated and described what are 

from the graphic controller 32 by setting the R,G,B output considered to be preferred embodiments of the present 

register to an ON state, and turns ON the HDD 81 if there invention, it will be understood by those skilled in the art 
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that various changes and modifications may be made, and 
equivalents may be substituted for elements thereof without 
departing from the true scope of the present invention. In 
addition, many modifications may be made to adapt a 
particular situation to the teaching of the present invention 
without departing from the central scope thereof Therefore, 
it is intended that the present invention not be limited to the 
particular embodiment disclosed as the best mode contem- 
plated for carrying out the present invention, but that the 
present invention includes all embodiments falling within 
the scope of the appended claims. 
What is claimed is: 

1. A computer system for controlling a display, the com- 
puter system having a central processing unit, a memory, a 
system bus and a display, comprising: 

a power controller for converting an operation mode of 
the computer system into a power management mode, 
said power management mode representing one of a 
power-on mode, a standby mode, a graphic off mode, 
and a suspend mode according to an access state of the 
computer system, said power controller for converting 
the computer system into said standby mode when 
power is applied to the computer system and there is no 
access to the computer system, for converting the 
computer system to said graphic off mode in which a 
screen-display state of said display is controlled 
according to input of a password, and for converting the 
computer system into said suspend mode when the 
computer system is not accessed for a predetermined 
time during said graphic off mode; and 

a screen controller for converting the screen display state 
of said display according to a designated said power 
management mode, and said screen controller for con- 
trolling the screen display state of said display accord- 
ing to input of a password when the computer system 
is accessed in said graphic off mode, the computer 
system checking an input password when entered by a 
user in said graphic off mode and, when the input 
password entered in said graphic off mode is a correct 
password, the computer system by said screen control- 
ler then controlling the screen display state to permit a 
visual display on said display. 

2. The computer system of claim 1, further comprising a 
timer for measuring a standby operation time when the 
computer system operates in said standby mode for a 
predetermined time, and said timer generating when said 
predetermined time in said standby mode has passed a 
graphic off signal to said screen controller to enter, said 
graphic off mode. 

3. The computer system of claim 2, further comprised of 
said screen controller controlling the screen display state of 
said display by the graphic off signal input from said timer, 
and said screen controller controlling the screen display state 
of said display by a graphic cut-off signal according to the 
input password. 

4. A method for controlling a screen display state of a 
computer system, comprising the steps of: 

converting the computer system into a standby mode 
when power is applied to the computer system and 
there is no access to the computer system; 

converting the screen display state of a display when there 
is no access to the computer system for a predetermined 
time during said standby mode, and converting the 
computer system to a graphic off mode in which the 
screen display state of said display is controlled accord- 
ing to input of a password when the computer system 
in said graphic off mode is again accessed; 
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checking by the computer system an input password when 
entered by a user in said graphic off mode and con- 
trolling the screen display state by the computer system 
to permit a visual display on said display when the 
input password entered in said graphic off mode is a 
correct password; and 

converting the computer system into a suspend mode 
when the computer system is not accessed for a pre- 
determined time during said graphic off mode, 

5. The method of claim 4, further comprised of the screen 
display state of the display being controlled by a graphic 
cut-off signal output from an input/output controller to a 
graphic controller. 

6. A method for controlling power supply to a computer 
system having a computer, a monitor and input devices, said 
method comprising the steps of: 

initializing said computer system upon activation of 
power; 

requesting entry of a password when the password is set 

at the time of a computer setup; 
when the password is entered correctly, supplying power 

to the computer system for use in a power-on mode of 

operation; 

determining whether there is activity from input devices 
indicating that the computer system is in use for a first 
time period; 

when there is activity from the input devices indicating 
that the computer system is in use during the first time 
period, maintaining the power supply to the computer 
system; 

when there is no activity from the input devices indicating 

that the computer system is not in use during the first 

time period, converting the computer system into a 

standby mode of operation; 
determining whether there is further activity from the 

input devices for a second time period; 
when there is activity from the input devices during the 

second time period, resupplying power back to the 

computer system and maintaining the power supply to 

the computer system for use; 
when there is no further activity from the input devices 

during the second time period, converting the computer 

system into a graphic-off mode of operation shutting off 

a visual display on the monitor; 
determining whether there is yet activity from the input 

devices for a third time period; 
when there is activity from the input devices during the 

third time period, requesting entry of said password 

before resupplying power back to the computer system 

for use in the power-on mode; 
when there is no activity from the input devices during the 

third time period, converting the computer system in a 

suspend mode of operation. 

7. The method of claim 6, further comprised of supplying 
power to the computer system for use in said power-on mode 
of operation without requesting entry of said password when 
the password is not set at the time of the computer setup. 

8. A computer system, comprising: 

a main computer containing a controller and auxiliary 
devices; 

a monitor physically separated from but electrically con- 
nected to said main computer; 
input devices connected to said main computer; and 
said controller controlling power supply and a screen 
display state of said monitor by the steps of: 
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initializing said computer system upon activation of 
power; 

requesting entry of a password when the password is 
set at the time of a computer setup; 

when the password is entered correctly, supplying 5 
power to the computer system for use in a power-on 
mode of operation; 

determining whether there is activity from input 
devices indicating that the computer system is in use 
during a first time period; 10 

when there is no activity from the input devices indi- 
cating that the computer system is not in use during 
the first time period, converting the computer system 
into a standby mode of operation; 

determining whether there is further activity from the is 
input devices for a second time period; 

when there is no further activity from the input devices 
during the second time period, converting the com- 
puter system into a graphic-off mode of operation 
shutting off a visual display on the monitor; 20 

determining whether there is yet activity from the input 
devices for a third time period; 

when there is activity from the input devices during the 
third time period, requesting entry of said password 
before resupplying power back to the computer 25 
system for use in the power-on mode; 

when there is no activity from the input devices during 
the third time period, converting the computer sys- 
tem in a suspend mode of operation. 



,962 

12 

9. The computer system of claim 8, further comprised of 
said controller supplying power to the computer system for 
use in said power-on mode of operation without requesting 
entry of said password when the password is not set at the 
time of the computer setup. 

10. The computer system of claim 8, further comprised of 
said controller maintaining the power supply to the com- 
puter system, when there is activity from the input devices 
indicating that the computer system is in use during the first 
time period. 

11. The computer system of claim 8, further comprised of 
said controller resupplying power back to the computer 
system and maintaining the power supply to the computer 
system for use, when there is activity from the input devices 
during the second time period. 

12. The computer system of claim 10, further comprised 
of said controller resupplying power back to the computer 
system and maintaining the power supply to the computer 
system for use, when there is activity from the input devices 
during the second time period. 

13. The computer system of claim 9, further comprised of 
said controller maintaining the power supply to the com- 
puter system, when there is activity from the input devices 
indicating that the computer system is in use during one of 
the first time period and the second time period. 

* * * * * 
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[57] ABSTRACT 

A continuous monitoring system for regulating access to a 
computer system or other restricted environment is dis- 
closed. The system employs real-time face recognition to 
initially detect the presence of an authorized individual and 
to grant the individual access to the computer system. In 
some embodiments, the system also employs real-time face 
recognition to continuously or periodically track the contin- 
ued presence of the authorized individual. Access to the 
computer system is revoked when the individual's presence 
is no longer detected. In other embodiments, the system 
employs a screen saver program to deny access to the 
computer system when a predetermined period of user 
inactivity is detected. Other aspects of the invention include 
a stranger detector which warns the authorized individual of 
the approach of an unauthorized individual, a multimedia 
messaging center which permits unauthorized individuals to 
leave messages for authorized individuals, and an adaptive 
enrollment program which permits the system to update the 
stored video images of authorized individuals to reflect the 
individuals' current appearance. 

48 Claims, 11 Drawing Sheets 
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CONTINUOUS VIDEO MONITORING USING monitors the identity of an authorized user and prevents 

FACE RECOGNITION FOR ACCESS access to a computer system without shutting the computer 

CONTROL system down when it is determined that the authorized user 

has left the computer unattended. 

COPYRIGHT AUTHORIZATION s \ n addition, the need for a continuous monitoring system 

A portion of the disclosure of this patent document exists not onl y with respect to virtual environments such as 

contains material which is subject to copyright protection. a computer system, but more generally extends to other 

The copyright owner has no objection to the facsimile restricted environments, including physical environments 

reproduction by anyone of the patent document or the patent suc ^ as ^ an ^ vau l ts - 

disclosure, as it appears in the Patent and Trademark Office OBJECTS AND SUMMARY OF THE 

patent file or records, but otherwise reserves all copyright INVENTION 
rights whatsoever. 

It is therefore an object of the present invention to provide 
FIELD OF THE INVENTION a security system which continuously monitors the identity 
Tbc invention relates to a system employing real-time " of , an aut h° rized ind ™ d(ia ' and prevents access to a coin- 
face recognition software to regulate and monitor access to £ u er syst ™ °' , h K • ? , ™— n } f w *™ } l . ,s 
computer! and other restricted environments. determined that the authorized mdiv.dual has left the vicin- 

ity of the environment. 

BACKGROUND OF THE INVENTION 2Q It is a further object of the present invention to continu- 
ously verify the identity of an authorized user in real time. 

Many computer systems permit access only to authorized Tt . A . . 4 c + u * • *• * _r 

„ -TT- - • ii i- i_ j l ■ 4 . It is a further object of the present invention to perform 

users. Inis is generally accomplished by requiring those t - \ . r *. . 

, . f? / 4 . iL • i the continuous monitoring in a manner which is passive, i.e., 

seeking to use the system to prove that they are authorized . • . , & . *• ■ «■ r , L 

. i -rj c c ii_ ■ *■ .i r rxc. m a manner which does not require the participation of the 

to do so. Proof or authorization can take many forms. Often, n r r 

the user must provide a name or initials and a password or 25 B ' . r L 

Personal Identification Number (PIN) before being permit- 11 B . a further ob J ect of lhe P resent invention to assign an 

ted to use the system. Other systems require the user to insert authorization level to each authorized individual and to 

a magnetic card or similar "key" into a reader which verifies P ermit each authorized individual access only to those 

that the cardholder is authorized to use the system. Still a PPacation programs and data appropriate for the user's 

others evaluate some biometric characteristic of the user, 30 autnonzatlon level - 

such as the user's voice print. It is a further object of the present invention to provide a 

All such access control systems, however, suffer from system which continuously monitors whether a 

several drawbacks. The most important is that they merely P er f° n ° ther than ^ »^onzed user has approached 

restrict initial access to the computer system. Once a user within reading distance of a display of the computer system 

has gained access, continued use of the system is possible by 35 * nd J"***' ^ on d l etectl ? n of a ™ance, d * ables lt f 

someone else when the authorized user leaves the computer or alerts the authonzed user of penon s 

system unattended. This decreases the security of the system presence. 

for several reasons. If the user leaves the computer It is a further object of the present invention to provide a 

unattended, an unauthorized user may gain access to sensi- continuous monitoring system which employs face recog- 

tive data stored in the computer system. Moreover, the 40 nition t0 passively regulate both initial and continuing 

unauthorized user might also be in a position to modify or access to a computer system. 

even erase data stored in the computer system. It is a further object of the present invention to fuse face 

To avoid this possibility, wary authorized users may recognition and speech recognition to provide swift initial 

choose to exit the system even when stepping away from the access to a computer system. 

computer for only a short time. This, too, has a drawback: It is a further object of the present invention to employ the 

the authorized user must reenter the computer system when screen saver utility resident in most operating systems to 

he returns, a process which may take several minutes. revoke access to a computer system when a significant 

Furthermore, this tactic does not permit continued running period of user inactivity is detected, 

of application programs such as spreadsheets while the user 5Q It is a further object of the present invention to employ the 

is away. As a result, users may end up "standing guard" at screen saver utility resident in most operating systems to 

their computers when running sensitive spreadsheets and launch a face recognition program which revokes access to 

other programs. a computer system when it fails to identify the continued 

Second, presently available access control systems do not presence of an authorized individual, 

offer convenient hands-off or passive operation, since they 55 It is a further object of the present invention to employ the 

require the active participation of the user before entry is screen saver utility resident in most operating systems to 

granted. Therefore, in addition to the inconvenience of launch a motion detection program which revokes access to 

remembering a password or of carrying a magnetic card, the computer system when it fails to detect motion in the 

users suffer the additional inconvenience of typing the vicinity of the computer system. 

password into a keyboard or swiping the magnetic card, 60 it is a further object of the present invention to provide a 

before access is granted. video monitoring system which detects the presence of an 

Furthermore, security systems that use passwords or cards unauthorized person, and upon detection of this 

can be compromised since a password can be discovered by circumstance, displays a greeting to that person and permits 

an intruder and cards can be stolen. the person to leave a message for an authorized person. 

A need therefore exists in the art for a security system 65 These and other objects of the invention are accomplished 

without the above drawbacks. In particular, there is a need by a system comprising a video input device coupled to a 

for a convenient passive security system which continuously general purpose computer or other specialized hardware 
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furnished with a face-recognition software program. The in the field of view of video camera 150 are authorized to 

face recognition algorithm is capable of identifying faces in have access to computer system 100. 

real time. The system repeatedly compares the face regis- Before describing the function of the continuous moni- 

tered by the video input device with the facial representa- tor ing system of the present invention, a short review of the 

tions of authorized individuals. When the comparison fails 5 principal steps in face recognition would be useful. As 

to indicate a match, continued access to the computer system sn0 wn in FIG. 2, face recognition can be broken down into 

is denied. a sequence of discrete tasks. In detection step 210, the face 

recognition system searches the field of view of a video 
input device for faces. 

The above objects and summary of the invention will be 10 In a real-time system — such as the system of the present 

better understood when taken in conjunction with the fol- invention — face detection must be accomplished in a frac- 

lowing detailed description and accompanying drawings in tion of a second. This is a challenge since the search area 

which: consists of the entire field of view of video camera 150 

FIG. lis a block diagram of a preferred computer system 5 which ^ often as lar S e as 640 bv 480 pixels. Current 

architecture implementing the continuous monitoring sys- algorithms meet this challenge and accomplish real-time 

tem of the present invention; detection by employing either a multiscale search strategy, 

no. 2 is a high level flowchart depicting the steps in face * ™f™ c sear( * strate S^ or both ' ^ ***** * he entin ; 

recognition* view t0 searcneci at a considerably higher speed 

- . ' „ , tnan would otherwise be possible. 

FIG. 3 is a flowchart depicting the not- tracking mode of 20 w . . , . , „ , r „ 

t . _ . • „ ™t-~. ~f Multiscale search algorithms initially search for faces in 

the continuous monitoring system or the present invention; , . . , . , 

. . . low resolution and switch to high resolution only when the 

FIG. 4 is a flowchart depicting the tracking mode of the low resolution sea rch indicates a head-like shape. Multicue 

continuous monitoring system of the present invention; search algorithms on the other hand, initially search for 

FIG. 5 is a flowchart depicting an inactivity detection easily detected cues which indicate the presence of a face, 

mode; For example, the presence of a face in the field of view often 

FIG. 6 is a flowchart depicting an alternative inactivity generates discontinuities in the spatial, temporal, and color 

detection mode; domains of the video image. Multicue algorithms search for 

FIG, 7 is a flowchart depicting a speech recognition such discontinuities and further examine only those areas 

sub-mode of the not-tracking mode shown in FIG. 3; 30 where the discontinuities are significant. Software programs 

op a . a i * j • * * .u *• c u- f° r performing real-time face detection using a multiscale 

rIG. 8 is a flowchart depicting the operation of a mulh- j » • . . „ , , 

. , uuj « ir fl nd multicue search strategy are commercially available, 

media messaging center which does not employ face rec- ~ . . . J , * j ,-■ jit j 

oenition* e su P 1 " 0 ^ 111 is. the C++ function CHead::FindHead- 

' Position of the Facelt Developer Kit. A copy of the object 

FIG. 9 is a flowchart depicting an enrollment program for code and a descri ption of the Application Programming 

storing the facial images of individuals authorized to have ^ interface (API) for this function is found in microfiche 

access to a restricted environment; and appendix A hereto. A person skilled in the art can use CHead 

FIG. 10 is a flowchart depicting an adaptive enrollment to perform face detection from video input as specified by 

program for updating the stored images of individuals the API. 

authorized to have access to a restricted environment. ^ 0nce a face has been dete cted, the face recognition 

DETAILED DESCRIPTION OF THE system performs alignment step 220 to precisely determine 

PREFERRED EMBODIMENTS ^ **f? P 05 *™' , size ' f d P 0S !* P 1 * Ste P rec * uir f es 

detailed shape and teature detection. Software programs for 

Referring to the drawings, FIG. 1 shows a computer performing alignment step 220 are also commercially avail- 
system 100 which comprises a CPU 110, a memory 120, and 45 able. One such program is the function CAlignment:: Find- 
input/output (I/O) means such as terminal 130. Memory 120 AlignedPosition of the Facelt Developer Kit. A copy of the 
stores application programs run on CPU 110 as well as other object code of this program is also found in microfiche 
data and information. appendix A hereto. 

Computer system 100 also comprises an image memory Next, the face recognition system performs normalization 
170, a face templates memory 140 and a video input device 50 step 230. Here, the head is normalized by scaling, rotating, 
such as video camera 150. As described more fully below, and warping, so that the face can be registered and mapped 
image memory 170 stores one or more facial images of each into a canonical size and pose regardless of its location and 
individual authorized to have access to computer system distance from video camera 150. Normalization is also 
100. When computer system 100 is initialized, the stored performed with respect to lighting variability. Normalization 
images of each authorized individual are converted to a 55 step 230, too, can be performed using commercially avail- 
facial representation or template. These templates are stored able software programs such as CFace::GetCanonicalImage 
in face templates memory 140. If desired, image memory function of the Facelt Developer Kit. A copy of the object 
170 and face templates memory 140 may be implemented as code of this program is also found in microfiche appendix A 
part of memory 120. Memory 120 also stores a real-time hereto. 

face recognition software program 160, 60 The fourth step is representation step 240. Every face 
In a preferred embodiment, the video camera 150 is recognition system utilizes an internal representation 
positioned such that a user sitting in front of terminal 130 scheme which it employs to translate facial data into a 
would be in the field of view of video camera 150. As unique coded characterization of the face of each individual, 
explained in more detail below, program 160 enables com- The representation scheme permits relatively simple com- 
puter system 100 to match facial images transmitted by 65 parison of acquired facial data with stored facial data to 
video camera 150 against facial representations stored in confirm the identity of a particular individual. A preferred 
face templates memory 140 and to ascertain whether persons embodiment for converting the images stored in image 
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memory 170 into templates suitable for performing face At this point, the system enters sub-mode three of the 

recognition is the function CFace::LoadFace of the Facelt not-tracking mode which comprises matching the acquired 

Developer Kit. A copy of the object code of this function facial representation against the stored facial representations 

may be found in microfiche Appendix A of this application, 0 f individuals authorized to use computer system 100. As 

In this preferred embodiment, when computer system 100 5 srj0 wn in FIG. 3, steps 335-350 comprise a loop which 

boots up, it retrieves the images of all authorized users from successively compares the acquired representation with each 

image memory 170 and converts them to face templates. The of the stored representations of authorized individuals until 

system then stores these templates in face templates memory a match is fouQd or ^ aJ1 of tfae stQred representations 

14U where they remain until computer system 1UU is shut haye been examined ^ noled above , the stored 

represen- 

' 4 ... T ■ * . t , 10 tations are generated from the images of authorized indi- 

. s e P 18 ma c ^ n 8 . s,e P 250 - J" th f , ste P ,he viduals stored in image memory 170 and are maintained in 

acquired facial representation is compared to at least one f , 5 1Aft J 

stored facial representation and a score is computed which taCe tem P lates memory 14U. 

represents the degree to which the acquired representation Continuing with FIG. 3, if no match is found in steps 
matches the stored representation. If the score is above a 335-350, the system returns to step 310 of sub-mode one. If, 
predetermined threshold, a match is declared, and the face in 15 on tne other hand, a match is found, decision step 340 
the field of view is identified as belonging to the person succeeds and the individual in the field of view of video 
associated with the stored faciaLrepresentation. Software camera 150 is granted access to computer system 100 as 
programs that generate a representation for faces, step 240, indicated in step 355. In one preferred embodiment, this 
and for performing matching step 250 on that representation § rant of access consists simply of enabling both the key- 
are commercially available. One such program is the func- 20 board and screen of terminal 130. In a second preferred 
tion CFace::IdentifyPerson of the Facelt Developer Kit. A embodiment, the grant of access may be tailored to the 
copy of the object code of this program is also found in authorization level of the individual. For example, a person 
microfiche appendix A hereto. with a particular authorization level might be granted access 

Of course, before a face recognition system can be used, t0 onl y certain data stored iQ memory 120 or might be 

authorized users must be enrolled into the database. A 25 Permitted to run only certain application programs, 

preferred embodiment of an enrollment program that is used It will be appreciated that the not- tracking mode described 

by the present invention to store the facial images of above provides completely passive access to computer sys- 

authorized individuals into image memory 170 is described tem 100. That is, access to computer system 100 is acquired 

below in connection with FIGS. 9 and 10. without the need to enter a password or other identifier, and 

The continuous monitoring system of the present inven- witn no need for a magnetic card or other "key." 

tion preferably comprises two modes: a not-tracking mode Once an individual has been granted access to computer 

and a tracking mode. These two modes are described in system 100, the system enters the tracking mode. In this 

connection with FIGS. 3 and 4. The continuous monitoring mode, the system continuously tracks the authorized indi- 

system depicted in FIGS. 3 and 4 is implemented in face 35 vidual and continues to permit access to computer system 

recognition software 160 which runs on CPU 110 of com- 100 only while the individual remains within the field of 

puter system 100. view of video camera 150. 

During periods when no one is detected in the field of In particular, once an individual is granted access to 

view of video camera 150, the system remains in not- computer system 100 in step 355 of FIG. 3, the system 

tracking mode. In this mode, the screen and keyboard of 40 immediately proceeds to step 410 of FIG. 4 where it 

terminal 130 are disabled, thus preventing access to the data registers the authorized individuars current head position, 

or application programs of computer system 100. shape, size, color and facial representation and stores this in 

The not-tracking mode may be divided into three sub- memory 120 as a new tracking path. The data stored in the 

modes. In the first sub-mode, which comprises steps tracking path can be generated using commercially available 

310-315 of FIG. 3, the system repeatedly searches for a 45 software such as the Facelt Developer Kit, a copy of which 

human face in the field of view of video camera 150. Thus, may be found in microfiche appendix A of this application, 

in step 310 the system searches the field of view of video This tracking path is used in subsequent searches to deter- 

camera 150 to determine whether it contains a human face. mine whether the authorized user remains in the field of 

This step corresponds to detection step 210 described above view of video camera 150. 

in connection with FIG. 2. 50 Specifically, in step 415 the system retrieves the current 

If a face is not detected in step 310, decision step 315 fails head location of the authorized user from memory 120 and 

and the system returns to step 310 to continue searching for searches for a face in the vicinity of that location. If a face 

a face in the field of view of video camera 150. is found, the system converts the newly acquired facial 

If a face is detected in step 310 (e.g., if an individual sits image to a facial representation and compares that repre- 

down to work at terminal 130 and thus enters the field of 55 sentation to the one stored in the tracking path. As is well 

view of video camera 150), decision step 315 succeeds and known in the art, this comparison may be performed through 

the system proceeds to sub-mode two of the not-tracking template matching using a normalized correlator. A match is 

mode. In sub-mode two, the system constructs a face tem- declared to exist if the normalized correlator is larger than a 

plate of the detected face. Thus, in step 325 the system preset threshold value. In a preferred embodiment, computer 

extracts the detected face from the video signal provided by 60 system 100 may also compare the newly acquired represen- 

video camera 150. This step corresponds to alignment step tation to the facial representations stored in the face template 

220 and normalization step 230 described above in connec- database. As described below, when these comparisons 

tion with FIG. 2. After alignment and normalization have sufficiently confirm the continued presence of the authorized 

been performed, the system proceeds to step 330 where it individual, continued access to computer system 100 is 

converts the facial image into a facial representation or 65 provided, 

template as described above in connection with representa- Thus, in decision step 420, the system determines whether 

tion step 240 of FIG. 2. the acquired representation matches the facial representation 
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stored in the tracking path. If decision step 420 succeeds, In an alternative embodiment, the tracking mode may 

then access to computer system 100 is continued, and the comprise a delay step 425a between step 425 and step 415. 

system proceeds to step 425 where the information stored in Delay step 425a causes the tracking mode to confirm the 

the tracking path is updated in accordance with the latest user's identity periodically rather than continuously. As will 

acquired representation. From step 425, the system loops 5 be recognized, this alternative places fewer demands on the 

back to step 415, and a new search is begun. In this way, the resources of computer system 100 but provides somewhat 

identity of the authorized user is repeatedly confirmed. decreased security. 

At times, however, decision step 420 may fail even when f f A . . 4 . . . . 

the authorized individual continues to sit before terminal In a P f£ erred embodimenl > ^ continuous monitoring 

130. This may happen, for example, if the individual looks 1fl T m of 1 _ l . h ^P J resent invention may comprise an additional 

down or away from the screen of terminal 130 (and thus is feature whlch ' durm S trackin S mode ' continuously searches 

not facing video camera 150) or if his facial features are fo / addltlonal individuals who have entered the field of view 

temporarily partially blocked. Therefore, as described of Vlde0 camera 150 11115 ma y for exam P le > if an 

below, when the system is unable to identify the facial unauthorized individual approaches the authorized indi- 

features of an individual in the field of view, it proceeds to c Vldual from behind and P°stf°ns himself to read data 

a second order identification scheme to confirm the continu- d *Play«J on the screen of terminal 130 over the shoulder of 

ing presence of the authorized individual the authorized individual. When equipped with this feature, 

Specifically, if decision step 420 fails, the system pro- the tnickin ? mode <*> m P ris * s ^ additional step 4256 which 

ceeds to decision step 430 where the system attempts to J^T^ searches for the presence of a second, unau- 

^ (i iU „ «■ ■ r*u «u • j • j- -j i thonzed individual within readme distance of the screen of 

confirm the continuing presence of the authorized individual ™ t . , , . j. , . \ «*■ 

on the basis of other recorded features such as head location, 2 ° tcrmmal 130 When such an individual is detected, the 

shape, color, and size which are stored as part of the tracking system e ? ther disable5 the screen or P rmts a warmn 8 mes " 

path. In a preferred embodiment, step 430 may be composed sage t0 the autnonzed USCL 

of two sub-steps. In the first sub-step, the system retrieves In an a lternative embodiment, the system employs the 

the most recent head-location of the authorized individual 25 not-tracking mode to regulate initial access to computer 

from the tracking path and determines whether the field of system 100, but substitutes an inactivity ^detection mode for 

view of video camera 150 now contains a head-shaped the backing mode of the preferred embodiment to regulate 

object in or near that location. If a head-shaped object is continuing access to computer system 100. In this altema- 

identified, the system proceeds to sub-step two and deter- tive embodiment, the system interprets prolonged inactivity 

mines whether other features of the detected head-shaped 30 b y tne authorized user as evidence that the authorized user 

object such as its shape, size and color, match the features has Ieft the vicinity of terminal 130. A preferred embodiment 

stored as part of the tracking path. A score is assigned to the of tne inactivity-detection mode of the present invention is 

results of this matching process, and if the score is above a described in connection with FIG. 5. 

predetermined threshold, decision step 430 succeeds and Recall that when an authorized user is recognized by the 

access to computer system 100 is continued. In that event, 35 not-tracking mode, access is granted to computer system 

the stored tracking path is updated in step 425, and the 100 (step 355 of FIG. 3) and the not-tracking mode termi- 

system returns to step 415 to repeat the tracking. nates. The system then proceeds to the inactivity-detection 

Otherwise, if sub-step one fails (i.e., no head shaped mode charted in FIG. 5 which employs the screen saver 

object is detected) or if sub-step two fails (i.e., the matching feature resident in most operating systems to revoke access 

score of the additional features is too low to indicate a 40 to computer system 100 and to return the system to the 

match), then step 430 fails. In that event, the system disables not-tracking mode. 

the keyboard and screen of terminal 130, and returns to step It is well known in the art that most operating systems 

310 of the not-tracking mode. In this way, the system comprise a screen saver feature. In accordance with this 

immediately revokes access when the presence of the autho- feature, when the operating system detects a preestablished 

rized individual ceases to be detected. 45 period of user inactivity (e.g., no keyboard activity), it 

In a preferred embodiment, because identification on the automatically launches a screen saver application program, 

basis of these additional features is less certain than facial Generally, the user may set the period's length and may 

identification, the system requires a closer spatial proximity choose the particular screen saver program to be launched, 

between the detected head-shaped object and the head As described below, this embodiment employs a particular 

location stored in the tracking path than would be required 50 screen saver program written specifically to achieve the 

to confirm a match on the basis of facial identification. This objects of the present invention. 

ensures the accuracy of the identification since it is impos- Specifically, in step 510 of FIG. 5, the operating system 

sible for an unauthorized individual to occupy the space of continuously monitors for a period of inactivity greater than 

the authorized individual within a single cycle of the track- the preestablished period. When such a period is detected, 

ing mode. 55 the system proceeds to step 515 wherein the operating 

It should be noted that in this preferred embodiment, the system launches the screen saver program. The screen saver 

continuous monitoring cycle represented by steps 415-430 program comprises code which directs computer system 100 

of the tracking mode is repeatedly executed as fast as the 10 disable the keyboard and screen of terminal 130 and to 

hardware will allow. On standard Pentium (™) hardware the return the system to not-tracking mode. Accordingly, in step 

time required to complete each monitoring cycle is appro xi- 60 520 access to computer system 100 is revoked, and the 

mately 10 to 20 milliseconds. As those skilled in the art will system returns to the not-tracking mode, 

appreciate, when other applications in addition to the con- In this way, access is denied to computer system 100 

tinuous monitoring program are being run on the CPU, during prolonged periods of inactivity by the authorized 

multitasking may be employed to execute the continuous user. This alternative embodiment is less expensive than the 

monitoring program and the other applications concurrently. 65 preferred embodiment described above and places fewer 

In some cases, this may somewhat increase the cycle time to, demands on the resources of computer system 100. It does 

for example, 30 milliseconds. not, however, immediately revoke access when the autho- 
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rized user leaves terminal 130 and therefore provides some- 
what less security than the continuous tracking of the 
preferred embodiment. 

In a related alternative embodiment, the screen saver 
program directs the system to examine the field of view of 5 
video camera 150 for the presence of the authorized indi- 
vidual before disabling the keyboard and screen of terminal 
130. This embodiment is described in connection with FIG. 
6. 

Steps 610 and 615 of this embodiment are identical to 10 
steps 510 and 515 of the embodiment charted in FIG. 5. 
Thus, in steps 610 and 615, the operating system continu- 
ously monitors the keyboard for a pre-set period of inactivity 
and launches the screen saver application when such a 
period is detected. 25 

In step 620, however, the screen saver program directs the 
system to search the field of view of video camera 150 for 
the face of the authorized individual initially granted access 
by the not-tracking mode. If the authorized individual is 
detected, decision step 625 succeeds, and continued access 20 
to the system is provided. Otherwise, decision step 625 fails, 
and the system proceeds to step 630 wherein access is 
revoked. From step 630, the system returns to the not- 
tracking mode. 

■ * 25 

It will be appreciated that this alternative embodiment 
beneficially maintains continued access despite long periods 
of inactivity by the authorized individual as long as the 
authorized individual remains at terminal 130. 

In a second related alternative embodiment, the system 3Q 
may maintain continued access despite prolonged inactivity 
as long as any motion is detected in the field of view of video 
camera 150. This embodiment is identical to that shown in 
FIG. 6, except that in steps 620 and 625 the system searches 
the field of view for any movement, rather than for the 35 
presence of the authorized individual. This embodiment 
avoids the need for performing face recognition in steps 620 
and 625 while providing continued access only when there 
is some indication that the authorized individual is still 
present. ^ ^ 

State of the art recognition software can compare an 
acquired representation against up to 300 stored represen- 
tations per second when run on a standard Pentium (™) 
processor. Therefore, when the number of authorized indi- 
viduals is small, the system can quickly determine whether 45 
a particular individual is authorized. 

When the number of authorized individuals is large, 
however, the time required to identify an individual may be 
substantial. Illustratively, in the present state of the art, the 
mean time to confirm the identity of an individual in a 50 
computer system with 25,000 authorized users is slightly 
over 40 seconds. 

Consequently, in a further preferred embodiment, the 
not-tracking mode additionally comprises a speech recog- 
nition sub-mode designed to decrease the number of com- 55 
parisons performed in steps 335-350 of FIG. 3. The speech 
recognition sub-mode is concisely indicated as step 330a in 
FIG. 3. A preferred embodiment of this sub-mode is more 
fully described in conjunction with FIG. 7. 

Recall that in step 330 of FIG. 3 the not-tracking mode 60 
creates an acquired facial representation of an individual in 
the field of view of video camera 150. In this preferred 
embodiment, before the comparison loop represented by 
steps 335-350 commences, the system proceeds to step 710 
wherein the individual is prompted to say his name or any 65 
other user specific phrase. In step 715, speech recognition 
software is employed to detect and identify the name spoken 
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by the individual. An example of speech recognition soft- 
ware suitable for this purpose is the Power Secretary by 
Articulate Systems. 

In step 720, the system retrieves from face template 
memory 140 the facial representations of all individuals 
whose names sound similar to the name spoken by the 
individual. The acquired representation is then compared 
only to the facial representations in this reduced universe. In 
this way, fast authorization determinations are possible on 
standard inexpensive hardware even when the universe of 
authorized individuals is large. 

It is noted that the preferred embodiment charted in FIG. 
3 employs face recognition to initially detect the presence of 
an individual. Speech recognition is used only to reduce the 
number of necessary comparisons. Alternatively, the system 
might instead employ speech recognition to initially detect 
the presence of an individual and launch the face recognition 
software only when an individual has been detected. 

In a further preferred embodiment, the not-tracking mode 
may comprise a surveillance feature which stores the facial 
representations of unauthorized individuals who approach 
terminal 130 during the absence of the authorized user. 
When equipped with this feature, the not-tracking mode is 
provided with an additional step 345a, shown in broken lines 
in FIG. 3. Recall that when decision step 345 succeeds, the 
system concludes that the individual in the field of view of 
video camera 150 is not an authorized individual. Then, in 
this preferred embodiment, the system proceeds to step 345a 
wherein the facial representation of the unauthorized indi- 
vidual is stored in a surveillance log in memory 120. Later, 
when the system detects the return of the authorized indi- 
vidual (step 340 of FIG. 3), it may display to the authorized 
individual a video image of any unauthorized individuals 
who approached terminal 130 during the authorized indi- 
vidual's absence. 

In a further preferred embodiment, the system can, while 
in not-tracking mode, serve as a multimedia messaging 
center. In this embodiment, when steps 335-350 of the 
not-tracking mode ascertain that a face in the field of view 
of video camera 150 does not belong to an authorized 
individual, the system plays a prerecorded multimedia greet- 
ing message on terminal 130 (step 3456). This message may 
contain both a visual portion displayed on the screen of 
terminal 130 as well as an audio portion if terminal 130 is 
provided with speakers. The greeting message offers the 
unauthorized individual the option of leaving a multimedia 
message for an authorized individual. In various preferred 
embodiments, the message may comprise a video compo- 
nent (using video camera 150), an audio component (using 
a microphone), a text component (using the keyboard of 
terminal 130), or any combination of the above. The entered 
message is stored by the system (step 345c). Later, when the 
system detects the return of the authorized individual (step 
340 of FIG, 3), it informs the individual of any messages 
received in the individual's absence and gives the individual 
the option to play back the messages. 

Aspects of this embodiment may be especially appropri- 
ate in securing certain physical environments, such as a 
family home. For example, a system embodying the not- 
tracking mode of the present invention in combination with 
the multimedia message center feature, could grant access to 
members of the family, while denying access to others. In 
addition, when a non-member of the family was recognized 
by the system, it could give the non-member the opportunity 
to leave a message for one or more of the family members. 

The preferred multimedia messaging center of the present 
invention profits greatly from its use of face recognition. 
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Specifically, face recognition permits the multimedia mes- Specifically, returning to FIG. 9, once an image has been 

saging center to automatically distinguish between autho- stored in image memory 170 (step 935), the system proceeds 

rized and unauthorized individuals and to display a greeting to decision step 940 wherein it is determined whether the 

to unauthorized individuals only. It should be recognized, desired number of stored images for this enrollee have been 

however, that the multimedia messaging center of the 5 acquired. If decision step 940 succeeds, then the system has 

present invention may be implemented without the use of stored tne desired number of images for this enrollee and the 

face recognition, as well. One such embodiment is shown in enrollment program terminates (step 945). 

FIG. 8. This embodiment is suitable, for example, in an . , . . . QAn e .« , 

a- . u • u i ii . j Otherwise, decision step 940 fails and the system returns 

office environment wherein each employee is allocated a PC , A1A ' _ , L1 J . c , 

for his orimarv use 10 P s 910-"25 to acquire another acceptable image of the 

As shown in FIG. 8, the multimedia messaging center 10 ™° n ™' ™ e , s y stem then P rocec f d * t0 de ™ Ste f ™ 

remains quiescent until, in step 810, it is restored by a visitor ^ hlch now fads since an image of me enrollee has already 

who enters a unique keystroke pattern via the keyboard of been s ' ored in im *f memor y 170 ^ s y stem therefore 

the PC (e.g., alt-m). When thus restored, the multimedia proceeds to step 950. 

messaging center proceeds to step 815 wherein it displays a The purpose of steps 950 and 955 is to ensure that the 

greeting to the visitor and offers the visitor the opportunity 15 second captured image is sufficiently different from the first 

to leave a message for the PC's primary user. In step 820, the captured image to justify its addition to the database. Thus, 

messaging center stores the message left by the visitor, and in step 950, the system converts both the second acquired 

displays a flag on the PC's monitor indicating the existence image and the first acquired image into templates and 

of a message. When the user returns, he may retrieve the compares the two. A high matching score indicates that the 

message. 20 two i ma g es are no t significantly distinct. In that event, 

A preferred embodiment of an enrollment program for decision step 955 succeeds and the second image is dis- 

storing the facial images of authorized users is now carded. Illustratively, the threshold score required to discard 

described. This preferred embodiment employs a software an m step 955 of piG. 9 might be the same as the score 

program which may be run only by a system administrator req uired to identify an individual in step 340 of FIG. 3. 

or someone with super-user privileges. The enrollment pro- 25 When dec ision step 955 succeeds, the system returns to step 

gram is described in connection with FIG. 9. 91Q tQ acquife anQther image of the enroUee 

Turning to FIG 9, in step 905 the system administrator If how ^ matchi ^ of ^ arison is low 

launches the enrollment program. In steps 910 and 915, the . , . . V f *i j «i_ j • jj j 

, i_ r*t.f f ii • then decision step 955 fails, and the second image is added 

system continuously searches for the face of an enrollee in . — . ... 4 , 

the field of view of video camera 150. Once a face is found, 30 '° ima | e memor y ,™ e P rocess 15 then re P ea * ed untl1 the 

decision step 915 succeeds and the system proceeds to step desired number of dissimilar images is stored in image 

920 wherein the detected face is extracted from the video memory 170. Once the desired number of images have been 

signal and displayed as an image on the monitor of terminal stored > decision step 940 fails, and the enrollment program 

130. Steps 910-920 may be performed using commercially terminates. 

available software such as CHead::FindHeadPosition, 35 Id a preferred embodiment, the enrollment program can 

CAlignment::FindAlignedPosition, and CFace::GetCanoni- comprise an adaptive enrollment scheme which periodically 

calmage of the Facelt Developer Kit. A copy of the object adds an updated image of the authorized individual to image 

code of these functions is found in microfiche Appendix A memory 170. This preferred embodiment is described in 

of this application. connection with FIG. 10. 

In step 925, the administrator is given the option of 40 Recall that when an authorized user is recognized by the 

discarding this acquired image. This option is provided not-tracking mode, access is granted to computer system 

because it has been found that many enrollees do not wish 100 (step 355 of FIG. 3) and the not-tracking mode termi- 

an unflattering image of themselves to be stored in memory. nates. In this preferred embodiment, the system then pro- 

If the system administrator rejects the image, then decision ceeds to step 1005 wherein it determines the amount of time 

step 925 fails and the system returns to step 905 to search 45 that has passed since an image of the authorized individual 

again for a face in the field of view of video camera 150. was added to image memory 170. If that amount of time 

Otherwise, step 925 succeeds and the system proceeds to exceeds a predetermined amount (which may be set by the 

decision step 930. There, the system determines whether the system administrator), decision step 1010 succeeds, and the 

captured image is the first image acquired for the enrollee. image of the authorized individual acquired in step 325 of 

If it is, decision step 930 succeeds and the system stores the 50 FIG - 3 is added to ima fi e memory 170 (step 1015). 

image in image memory 170 (step 935). Otherwise, decision step 1010 fails, and the enrollment 

As those skilled in the art will recognize, it is possible to program terminates (step 1020). 

construct a face template from a single image of an indi- In this way, image memory 170 is periodically updated to 

vidual. It is preferable, however, to employ two or more reflect changes in the appearance of the authorized indi- 

images in constructing the template since this yields a more 55 vidual. As a result, the system can continue to recognize the 

refined template and commensurately more accurate face authorized individual even as his appearance changes over 

recognition. time, 

In addition, in order to achieve a substantial refinement in Preferably, the initial images stored at the time of enroll- 

the template, it is important that the plurality of images from ment are never erased, but the additional images added 

which the template, is derived be substantially dissimilar. 60 periodically may be replaced during subsequent periodic 

Otherwise, the marginal information content added by each updates. 

additional image is small and does not significantly improve It should be recognized that this preferred adaptive enroll- 

the quality of the template. For this reason, the preferred ment embodiment conveniently updates the stored images of 

enrollment embodiment of the present invention discards an authorized individual without requiring the individual to 

additional captured images of the enrollee unless they differ 65 participate in a new enrollment procedure, 

substantially from those images already stored in image While the invention has been described in conjunction 

memory 170. with specific embodiments, it is evident that numerous 
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alternatives, modifications, and variations will be apparent 
to those skilled in the art in light of the foregoing descrip- 
tion. 

What is claimed is: 

1. A method of regulating continued access to a restricted 
environment, comprising: 

storing a facial representation of an individual authorized 
to have access to the restricted environment; 

periodically acquiring a facial representation of an indi- 
vidual desiring continued access to the restricted envi- 
ronment; 

determining repeatedly whether the ■ individual seeking 
continued access is the authorized individual by com- 
paring the most recently acquired representation to the 
stored representation to determine the degree to which 
the most recently acquired representation corresponds 
to the stored representation; 

storing additional identifying features representative of an 
authorized individual; 

acquiring additional identifying features of the individual 
desiring continued access to the restricted environment; 

comparing the acquired additional features and the stored 
additional features when the degree of similarity 
between the most recently acquired facial representa- 25 
tion and the stored facial representation does not con- 
firm the identity of the individual seeking continued 
access; 

revoking access to the restricted environment if the step of 
determining and the step of comparing indicate that the 
individual seeking continued access is not the autho- 
rized individual. 

2. The method of claim 1 wherein the additional identi- 
fying features comprise the head location, head shape, and 
head size of the authorized individual. 

3. The method of claim 1 wherein the stored representa- 
tion is modified by at least a portion of the acquired 
representations. 

4. The method of claim 1 further comprising: 
determining if no individual is seeking access to the 

restricted environment; and 
revoking access to the restricted environment if the deter- 
mination indicates that no individual is seeking access 
to the restricted environment. 

5. The method of claim 1 wherein the restricted environ- 
ment is a physical environment. 

6. The method of claim 1 wherein a video camera is 
employed to acquire a facial image of the individual seeking 
access to the restricted environment. 

7. The method of claim 1 wherein the comparison is 
performed in real-time. 

8. The method of claim 1 wherein the step of acquiring a 
facial representation comprises the steps of: 

detecting the presence of a face in a field of view; 
aligning the detected face; 
normalizing the detected face; and 
representing the detected face as a template. 

9. The method of claim 1, further comprising: 
monitoring the environment for periods of inactivity 

greater than a predetermined length; and 
revoking access to the environment when such period of 
inactivity is detected. 

10. The method of claim 1 further comprising a method of 65 
regulating initial access to the restricted environment, com- 
prising: 
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storing facial representations of individuals authorized to 
have access to the restricted environment; 

acquiring a representation of the face of an individual who 
approaches the restricted environment during a period 
when access to the restricted environment is impeded; 

comparing the initial acquired representation to the stored 
representations; and 

denying initial access to the restricted environment if the 
comparison indicates that the individual seeking initial 
access is not an authorized individual. 

11. The method of claim 10 further comprising: 
discerning the presence of a face in the field of view of a 

video camera. 

12. The method of claim 1 wherein the restricted envi- 
ronment is a virtual environment. 

13. The method of claim 12 wherein the virtual environ- 
ment is a computer network environment, 

14. The method of claim 12 wherein the virtual environ- 
ment is a PC environment. 

15. A method of regulating initial and continued access to 
a restricted environment, comprising: 

storing facial representations of individuals authorized to 
have access to the restricted environment; 

acquiring a first representation of the face of an individual 
who approaches the restricted environment during a 
period when access to the restricted environment is 
impeded; 

comparing the first acquired representation to the stored 
representations; 

denying initial access to the restricted environment if the 
comparison indicates that the individual seeking initial 
access is not an authorized individual and granting 
initial access to the restricted environment if the com- 
parison indicates that the individual seeking initial 
access is an authorized individual; 

acquiring a second facial representation of an individual 
desiring continued access to the restricted environment; 

determining whether the individual seeking continued 
access is the authorized individual by comparing the 
second acquired representation to at least one of the 
stored representations; 

revoking access to the restricted environment if the deter- 
mination indicates that the individual seeking contin- 
ued access is not the authorized individual; 

if initial access is denied, displaying a greeting to the 
unauthorized individual; and 

storing a message for an authorized individual from the 
unauthorized individual. 

16. The method of claim 15 wherein the message com- 
prises a text portion. 

17. The method of claim 15 wherein the message com- 
prises an audio portion. 

18. The method of claim 15 wherein the message com- 
prises a visual portion, 

19. A method of regulating continued access to a restricted 
environment, comprising: 

storing a facial representation of an individual authorized 
to have access to the restricted environment; 

acquiring a facial representation of an individual desiring 
continued access to the restricted environment; 

determining whether the individual seeking continued 
access is the authorized individual by comparing the 
acquired representation to the stored representation; 

revoking access to the restricted environment if the deter- 
mination indicates that the individual seeking contin- 
ued access is not the authorized individual; 



10/28/2003, EAST version: 1.4.1 



6,111,517 



15 



16 



20 



25 



30 



detecting the approach of an unauthorized second indi- 
vidual to the restricted environment; and 

revoking access of the authorized individual to the 
restricted environment when the approach of the unau- 
thorized individual is detected. 5 

20. A method of regulating initial access to a restricted 
environment, comprising: 

storing a plurality of facial representations of individuals 

authorized to have access to the restricted environment; 

.10 

associating one or more words with each authorized 
individual; 

acquiring a representation of the face of an individual who 
approaches the restricted environment during a period 
when access to the restricted environment is impeded; 15 

recognizing the linguistic content of the one or more 
words when spoken by an individual seeking access to 
the restricted environment; 

comparing the acquired representation only against the 
facial representations of individuals with whom the 
recognized one or more words are associated; 

denying initial access to the restricted environment if the 
comparison indicates that the individual seeking initial 
access is not an authorized individual. 

21. A continuous monitoring system for regulating access 
to a restricted environment, comprising: 

a CPU; 

a memory connected to the CPU; 

a tracking path stored in the memory comprising data 
regarding an individual authorized to have access to the 
restricted environment; 

the tracking path comprising at least a facial representa- 
tion of the authorized individual; 

a video input device connected to the CPU and having a 35 
field of view; 

an image translator resident in the CPU for repeatedly 
receiving images in the field of view and generating 
therefrom data regarding an individual located in the 
field of view; 

a comparator resident in the CPU for repeatedly compar- 
ing the generated data with the tracking path, the results 
of each comparison constituting a comparison result; 

an access control device resident in the CPU operative in 
response to the comparison results; 

wherein the generated data comprises a facial represen- 
tation of the individual in the field of view when it is 
possible to acquire a facial representation of the indi- 
vidual; 

wherein the tracking path and the generated data further 
comprise additional identifying data regarding the 
authorized individual located in the field of view; and 

wherein the comparator compares the additional gener- 
ated data and the additional tracking path data when it 
is impossible to acquire a facial representation of the 
individual in the field of view. 

22. The system of claim 21 wherein the generated data 
comprises a facial representation of the individual in the 
field of view. 

23. The system of claim 21 wherein the tracking path 
comprises the most current data available regarding the 
authorized individual. 

24. The system of claim 21 wherein the access control 
device denies access to the restricted environment when the 
comparison result indicates the absence of the authorized 
individual. 
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25. The system of claim 21 wherein the image translator, 
comparator, and access control device operate in real-time. 

26. The system of claim 21 wherein the additional data 
comprise the head location, head shape, and head size of the 
authorized individual. 

27. The system of claim 26 wherein the current data is at 
least partially derived from the additional generated data. 

28. A monitoring system for regulating continued access 
to a restricted environment, comprising: 

means for storing a facial representation of an individual 
authorized to have access to the restricted environment; 

means for acquiring a facial representation of an indi- 
vidual desiring continued access to the restricted envi- 
ronment; 

means for repeatedly determining whether the individual 
seeking continued access is the authorized individual 
by comparing the acquired representation and the 
stored representation; 

means for revoking access to the restricted environment if 
the determination indicates that the individual seeking 
continued access is not the authorized individual; 

second means for storing additional identifying features 
representative of an authorized individual; 

second means for acquiring additional identifying fea- 
tures of an individual desiring continued access to the 
restricted environment; 

wherein the means for determining comprises a means for 
comparing the acquired additional features and the 
stored additional features when the degree of similarity 
between the most recently acquired facial representa- 
tion and the stored facial representation does not con- 
firm the identity of the individual seeking continued 
access. 

29. The system of claim 28 further comprising: 
means for periodically acquiring subsequent facial repre- 
sentations of at least one individual desiring continued 
access to the restricted environment; 

means for repeatedly determining the degree to which the 
most recently acquired representation corresponds to 
the stored representation of the authorized individual. 

30. The system of claim 28 wherein the additional iden- 
tifying features comprise the head location, head shape, and 
head size of the authorized individual. 

31. The system of claim 28 wherein the stored represen- 
tation is modified by at least a portion of the acquired 
representations used to identify the individual desiring con- 
tinued access to the restricted environment, 

32. The system of claim 28 further comprising means for 
determining if no individual is seeking access to the 
restricted environment; and 

means for revoking access to the restricted environment if 
the determination indicates that no individual is seeking 
access to the restricted environment. 

33. The continuous video monitoring system of claim 28 
wherein the restricted environment is a physical environ- 
ment. 

34. The system of claim 28 wherein the means for 
acquiring comprises a video camera. 

35. The system of claim 28 wherein the means for 
comparing operates in real-time. 

36. The system of claim 28 wherein the means for 
acquiring a facial representation comprises: 

means for detecting the presence of a face in a field of 
view; 

means for aligning the detected face; 
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means for normalizing the detected face; and 

means for representing the detected face as a template. 

37. The system of claim 28, further comprising: 
means for monitoring the environment for periods of 

inactivity greater than a predetermined length; and 
means for revoking access to the environment when such 
period of inactivity is detected. 

38. The system of claim 28 further comprising means for 
regulating initial access to the restricted environment, com- 
prising: 

means for storing facial representations of individuals 
authorized to have access to the restricted environment; 

means for acquiring a representation of the face of an 
individual who approaches the restricted environment 
during a period when access to the restricted environ- 
ment is impeded; 

face recognition means for comparing the initial acquired 
representation to the stored representations; and 

means for denying initial access to the restricted environ- 
ment if the comparison indicates that the individual 
seeking initial access is not an authorized individual. 

39. The system of claim 38 wherein the means for 
acquiring comprises: 

a video input device having a field of view, and 
means for discerning the presence of a face in the field of 
view. 

40. The system of claim 28 wherein the restricted envi- 
ronment is a virtual environment. 

41. The system of claim 40 wherein the virtual environ- 
ment is a computer network environment. 

42. The system of claim 41 wherein the virtual environ- 
ment is a PC environment. 

43. A monitoring system for regulating initial and con- 
tinued access to a restricted environment, comprising: 

means for storing facial representations of individuals 
authorized to have access to the restricted environment; 

means for acquiring a first representation of the face of an 
individual who approaches the restricted environment 
during a period when access to the restricted environ- 
ment is impeded; 

face recognition means for comparing the first acquired 
representation to the stored representations; 

means for denying initial access to the restricted environ- 
ment if the comparison indicates that the individual 
seeking initial access is not an authorized individual 
and for granting initial access to the restricted environ- 
ment if the comparison indicates that the individual 
seeking initial access is an authorized individual; 

means for acquiring subsequent facial representations of 
an individual desiring continued access to the restricted 
environment; 

means for repeatedly determining whether the individual 
seeking continued access is the authorized individual 
by comparing the subsequent acquired representations 
and at least one of the stored representations; 
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means for revoking access to the restricted environment if 
the means for determining determines that the indi- 
vidual seeking continued access is not the authorized 
individual; 

means, operative upon detection of an unauthorized 
individual, for displaying a greeting to the unauthorized 
individual and message means for enabling the unau- 
thorized individual to leave a message for an authorized 
individual. 

44. The system of claim 43 wherein the message means 
comprises means for leaving a typewritten message. 

45. The system of claim 43 wherein the message means 
comprises means for leaving a spoken message. 

46. The system of claim 43 wherein the message means 
comprises means for leaving a message comprising both 
audio and visual components. 

47. A monitoring system for regulating continued access 
to a restricted environment, comprising: 

means for storing a facial representation of an individual 
authorized to have access to the restricted environment; 

means for acquiring a facial representation of an indi- 
vidual desiring continued access to the restricted envi- 
ronment; 

means for repeatedly determining whether the individual 
seeking continued access is the authorized individual 
by comparing the acquired representation and the 
stored representation; 

means for revoking access to the restricted environment if 
the determination indicates that the individual seeking 
continued access is not the authorized individual; 

means for detecting the approach of an unauthorized 
second individual to the restricted environment; and 

means, triggered by detection of the unauthorized second 
individual, for revoking access of the authorized indi- 
vidual to the restricted environment. 

48. A system for regulating initial access to a restricted 
environment, comprising: 

means for storing a plurality of facial representations of 
individuals authorized to have access to the restricted 
environment; 

means for associating one or more words with each 
authorized individual; 

means for acquiring a representation of the face of an 
individual who approaches the restricted envioronment 
during a period when access to the restricted environ- 
ment is impeded; 

means for recognizing the linguistic content of the one or 
more words when spoken by an individual seeking 
access to the restricted environment; 

means for comparing the acquired representation only 
against the facial representations of individuals with 
whom the one or more words are associated; and 

means for denying intial access to the restricted environ- 
ment if the comparsion indicates that the individual 
seeking initial access is not an authorized individual. 



10/28/2003, EAST Version: 1.4.1 



UNITED STATES PATENT AND TRADEMARK OFFICE 
CERTIFICATE OF CORRECTION 

PATENT NO. : 6,111,517 

DATED August 29, 2000 

INVENTOR(S) : Aticketal. 

It is certified that errors appear in the above-identified patent and that said Letters Patent is hereby 
corrected as shown below: 

after the Title of the Invention but before the Copyright Authorization: 

insert - This application contains a microfiche appendix. The microfiche appendix includes 22 
microfiche and 2070 frames. - 



Signed and Sealed this 
Fifteenth Day of May, 2001 

NICHOLAS P.CODICI 

Attesting Officer Aviing Director of the United States Patent and Trademark Office 



10/28/2003, EAST version: 1.4.1 



(12) United States Patent 

Lignoul 



US006374145B1 

(10) Patent No.: US 6,374,145 Bl 
(45) Date of Patent: Apr. 16, 2002 



(54) PROXIMITY SENSOR FOR SCREEN SAVER 
AND PASSWORD DELAY 

(75) Inventor: Mark Lignoul, 717 Oak La., 
Grapevine, TX (US) 76051 

(73) Assignee: Mark Lignoul, Grapevine, TX (US) 

( * ) Notice: Subject to any disclaimer, the term of this 
patent is extended or adjusted under 35 
U.S.C. 154(b) by 0 days. 

(21) Appl. No.: 09/211,091 

(22) FUed: Dec. 14, 1998 

(51) Int. CI. 7 G05B 15/00; G06F 11/30; 

G06F 19/00 

(52) U.S. CI 700/17; 700/83; 700/302; 

703/23; 713/200; 713/203; 345/867 

(58) Field of Search 700/17, 59, 66, 

700/83, 306, 65, 84, 85, 302; 348/156, 
154, 155; 713/202, 323; 703/21, 23, 24, 
26; 345/158, 867; 708/135; 710/18 

(56) References Cited 

U.S. PATENT DOCUMENTS 



4,433,309 
4,449,122 
4,454,594 
4,682,159 
4,685,056 
4,698,748 
4,742,191 
4,825,209 
4,886,941 
5,019,804 
5,059,961 
5,063,306 
5,270,710 A 
5,315,884 A 
5,329,471 A 
5,377,269 A 
5,380,983 A 
5,396,443 A 



2/1984 Hermle et al 331/65 

5/1984 Whitmer 340/562 

6/1984 Heffron et al 713/200 

7/1987 Davison 345/158 

8/1987 Barnsdale, Jr. et al 711/164 

10/1987 Juzswik et al 713/322 

5/1988 Coleman et al 200/52 R 

4/1989 Sasaki et al 340/825.72 

12/1989 Davis et al 178/19.01 

5/1991 Fraden 340/562 

10/1991 Cheng 345/10 

11/1991 Edwards 327/77 

12/1993 Gaultier et al 341/33 

5/1994 Kronberg 73/862.68 

7/1994 Swoboda et al 703/13 

12/1994 Heptig et al 713/202 

1/1995 Cavada et al 219/250 

3/1995 Mese et al 713/321 



5,404,541 A 4/1995 Hirosawa et al 713/324 

(List continued on next page.) 

OTHER PUBLICATIONS 

Microsoft Press Computer Dictionary 1997, Microsoft 
Press, third edition, p. 272.* 

Article entitled, "The Universal Serial Bus from Abstraction 
to Implementation", by Mohammed Fennich, Intel Corpo- 
ration. 

IBM Personal System/2 Mouse Technical Reference, second 
edition, Jun. 1989, IBM Corporation. 
Data Handbook IC20, 80C51— Based, 8-Bit Microcontrol- 
lers, Aug. 1996, Phillips Semiconductors. 



Primary Examiner — William Grant 
Assistant Examiner — Paul Rodriquez 
(74) Attorney, Agent, or Firm — J. 
Hunton & Williams 



Robert Brown, Jr.; 



(57) 



ABSTRACT 



Apparatuses and methods of the present invention provide 
improved control over the activation and deactivation of a 
computer program. For example, the computer program may 
be a screen saver program and/or a password protection 
program. The computer program is prevented from being 
activated when the user has not recently provided input to a 
computer system via an input-output device, but the user is 
still in the vicinity of the computer system. When no person 
is detected in the vicinity, the present invention allows the 
computer program's activation to be controlled using stan- 
dard known methods. In one embodiment, the present inven- 
tion provides enhanced screen saver activation and deacti- 
vation control without the need to modify existing computer 
hardware or software. Embodiments of modified computer 
hardware and software systems are also provided where it is 
desirable to integrate enhanced screen saver activation and 
deactivation control into a hardware or software system. 
Other aspects of the present invention deal with program 
activation control, user authentication, and methods for 
providing the most economical implementations. 

20 Claims, 3 Drawing Sheets 
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PROXIMITY SENSOR FOR SCREEN SAVER 
AND PASSWORD DELAY 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates generally to computer systems. 
More particularly, the invention relates to a proximity sensor 
based control system used to prevent a computer program 
such as a screen saver and/or a password protection program 
from being activated while an operator remains present in 
the vicinity of a computer. 

2. Description of the Related Art 

Screen savers are well known in the art of computer 
systems. Desktop computers such as personal computers and 
workstations use screen savers to display an image or 
animation after a period of user inactivity. Likewise, laptop 
computers and some personal digital assistants (PDAs) also 
employ screen saver . technology. Screen savers are 
employed to prevent long-term damage which would oth- 
erwise be experienced by a computer display monitor if a 
single pattern such as a menu interface to an operating 
system were constantly displayed. Computer programs 
designed for use in office environments commonly display a 
password dialog box as a part of the screen saver image or 
as a stand-alone function. Password protection advanta- 
geously safeguards sensitive information held within a com- 
puter system during intermediate periods of time when the 
user has left the work area. Password protection with user 
inactivity activation allows the worker to leave the work 
area without needing to log out of a session with the 
computer. This increases security, because the user will 
often neglect to log out of the computer when leaving the 
work area to attend a meeting, eat lunch or perform other 
tasks away from the work area. On the other hand, the user 
may desire to log out of the computer session at the end of 
the workday. 

Screen savers and password protection programs are 
typically activated by sensing a period of user inactivity. For 
example, such programs may be activated after a period of 
time has elapsed as measured from the last time the user has 
provided input to the computer using a keyboard and/or a 
mouse. One common method of controlling computer pro- 
gram activation is to use a timer. The timer is reset when an 
input is detected from the keyboard or the mouse. The timer 
is essentially a counter which counts for a predetermined 
number of clock cycles. After the timer has reached a 
predetermined count, the timer is said to "time-out." 
Typically, the timer generates a "time-out signal" upon 
time-out. The amount of time required for the timer to 
produce the time-out signal is called a "time-out period." 
Often the time-out signal is coupled to provide an interrupt 
to a central processing unit within the computer. When the 
time-out signal is detected, an interrupt request signal is 
provided to the central processing unit which in turn acti- 
vates the screen saver and/or password protection program. 
Once the timer begins counting, if the user provides another 
input via the keyboard or the mouse, the timer is reset and 
begins counting anew. Thus the screen saver and/or pass- 
word protection program will not be activated when the user 
is actively working with the computer. The screen saver 
and/or password protection program will only be activated 
after the user has remained inactive for the duration of the 
prespecified time-out period. The same holds true for other 
user activity controlled programs such as password protec- 
tion programs and on-line connection programs. An on-line 
connection program is a program which initiates and/or 



'4,145 Bl 

2 

deactivates a communication connection. For example, 
when a user has not sent any input to an on-line connection 
port for the duration of a time-out period, the on-line 
connection program will deactivate the connection so as to 

5 free up the connection port. 

User activity controlled program technology as currently 
implemented in the art has some attendant problems. Users 
often become annoyed at the side effects inherent in the 
existing technology. For example, in the case of screen 

10 savers, while the user is actively working within his or her 
work area, several interruptions may occur which divert the 
user's attention from the computer's keyboard or mouse. In 
some instances the user may be involved with paperwork, a 
telephone call or a face-to-face interaction with a customer 

15 or colleague. In all such cases, time-out period may elapse 
and cause the screen saver to be activated. When the user 
returns to the computer desiring to interact with the oper- 
ating system's user interface, the user is instead faced with 
a screen saver. In cases where the user needs to reenter a 

20 password, this is more than a mild annoyance. The password 
feature is useful in protecting the user's sensitive data when 
the user has left the work area for an extended period, but is 
more of a nuisance when the user has merely turned away to 
attend to another task such as a telephone call. 

25 Hence there is a need for a user activity controlled 
program technology which can control the activation of a 
computer program without the aforementioned problems. 
For example, it would be desirable to have a screen saver 
which could provide a screen saving functionality without 

30 being activated while a user remains within the work area. 
It would be desirable for the computer to detect the physical 
presence of the user, and to use this detection to prevent a 
screen saver from activating. It would be desirable to include 
new timer algorithms adapted for use with this detection. 

35 Moreover, it would also be beneficial to provide intelligent 
screen saver activation control without the need to modify 
existing computers, screen saver software, or operating 
systems. It would also be desirable to integrate such a screen 
saver into computer hardware and software in order to 

40 provide efficient and ergonomic implementations. Systems 
and techniques are also needed to provide for low cost 
solutions as well as systems with extended functionality. 
Moreover, it would be desirable to have a user activity 
controlled program technology which could be used to 

45 activate various types of programs other than screen saver 
programs. Such a technology could be used, for example, to 
control the activation of password protection programs and 
online connection control programs. 

50 SUMMARY OF THE INVENTION 

The present invention solves these and other problems by 
providing methods and apparatus which detect the physical 
presence of a user and use this detection to control the 
activation of a user activity controlled program such as a 

55 screen saver or a password program. In a first aspect, the 
present invention involves an interface system which can be 
coupled to a computer. This system is operative to detect the 
physical presence of a user via a proximity sensor and to 
transmit a signal indicative of the presence of the user to 

60 prevent a computer program from being activated. The 
proximity sensor is operatively coupled to this interface 
system. The interface system includes a control module such 
as a microcontroller which causes information to be trans- 
mitted based on an output signal provided by the proximity 

65 sensor to prevent the activation of the computer program. 
The transmitted information is preferably embodied as a 
signal containing at least one bit of information designated 
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by a Logic-one voltage level, a logic-zero voltage level, or a 
transition therebetween. A typical value for logic -one is five 
volts and a typical value for logic-zero is zero volts. 
Typically, the transmitted information includes a sequence 
of bit values to form a coded command signal indicative of 5 
a mouse movement or a keyboard stroke. In one 
embodiment, this apparatus is supplied as an interface 
system which may be interposed between a computer port 
such as a mouse port and an input device such as a mouse. 
The present invention may be used to control various types 3Q 
of computer programs such as screen saver programs and/or 
password programs. 

In a second aspect of the present invention, a timer is 
included in the interface system. When a timer is used, the 
interface system is operative to monitor a signal supplied by 15 
a user input -output device and monitor a signal supplied by 
a proximity sensor. The proximity sensor provides a signal 
indicative of the physical presence of a user. The system 
allows a computer program to be placed in a foreground 
state after a prespecified time duration has passed since both 2Q 
the user-input signal and the proximity-sensor signal have 
been inactive. 

A third aspect of the present invention involves methods 
of deactivating a computer program. In a first step, the 
physical presence of a user is detected via a first proximity 2 $ 
sensor. In a second step, the user is identified using at least 
one other proximity sensor. In a third step, a control signal 
is generated indicating to deactivate the computer program 
when the user has been identified as an authorized user. In 
one embodiment, the computer is a screen saver program 30 
with password protection, and in another embodiment the 
computer program is a stand-alone password protection 
program. 

A fourth aspect of the present invention involves a soft- 
ware system for use on computers and related computerized 35 
equipment. The software system involves a background- 
processing module coupled to receive inputs from a user 
input-output device and a proximity sensor. The software 
system also includes a foreground-processing module which 
is activated in response to a control signal generated by the 40 
background-processing-module. The background- 
processing module maintains a timer indicative of the 
amount of time since a user has last applied an input via the 
user input-output device. The background module also 
accepts an input from the proximity sensor and causes the 45 
foreground-processing module to be activated. The fore- 
ground process is activated when both the timer indicates no 
user input has been received via the user input-output device 
for a prescribed amount of time and the proximity sensor 
indicates no user is present. 50 

BRIEF DESCRIPTION OF THE FIGURES 

The various novel features of the invention are illustrated 
in the figures listed below and described in the detailed 
description which follows. 55 

FIG. 1 is a block diagram representing a computer system 
equipped with a proximity sensor and related control and 
interface apparatus. 

FIG. 1A is a block diagram illustrating an embodiment of 
a proximity sensor interface which can be connected to a 60 
standard computer mouse port. 

FIG, 2 is a block diagram representing a software archi- 
tecture of an operating system designed to provide a prox- 
imity sensor-based input-output operations. 

FIG. 3 is a flow chart illustrating a method of processing 65 
employed to provide a proximity sensor based computer 
program. 
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FIG. 4 is a flow chart illustrating a method of processing 
whereby a timer is employed to provide intelligent control of 
the activation or delay of a computer program. 

FIG. 4A is a flow chart illustrating a method of processing 
whereby a first timer is employed to provide intelligent 
control of the activation of a screen saver program, and a 
second timer is employed to deactivate a screen saver 
program. 

FIG. 5 is a flow chart illustrating a method of processing 
employed to provide enhanced services and security based 
on the use of a proximity sensor. 

FIG. 6 is a flow chart illustrating a method of sensor signal 
processing used to allow a low cost proximity sensor to be 
used to implement a reliable overall system. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENT 

In the detailed description of the preferred embodiment of 
the present invention which follows, specific embodiments 
are presented for purposes of illustration and description. 
While the present invention may be applied to control the 
activation of various types of computer programs, specific 
embodiments involving screen saver programs are provided 
herein merely by way of example. Timer controlled pro- 
grams such as screen savers, password protection programs 
and on-line connection programs fall under the broader 
category of "user activity controlled programs." For 
example, in embodiments disclosed herein, the "screen 
saver" program may be substituted by or augmented with 
any user activity controlled program such as a password 
protection program or an on-line connection program. Many 
modifications and variations of the present invention will be 
apparent to those of ordinary skill in the art. Therefore, it is 
to be understood that the present invention encompasses all 
such equivalent embodiments. 

FIG. 1 is a block diagram representing an embodiment of 
computer system 100 equipped with a proximity sensor and 
related control and interface apparatus. The computer sys- 
tem 100 includes a standard computer 104 which may be 
designed in accordance with the prior art. The standard 
computer 104 may be any workstation, personal computer, 
laptop computer, personal digital assistant, or other comput- 
erized apparatus which does not include or is not coupled to 
the proximity-sensor interface system 102. Once the 
proximity-sensor interface system 102 is interfaced to the 
standard computer 104, the computer system 100 results. 

The proximity-sensor interface system 102 includes a 
proximity sensor 105 operatively coupled to a microcon- 
troller 110. The proximity sensor 105 may be any device 
which indicates the physical presence of a user. The micro- 
controller 110 is controllably coupled to a control logic 
program 115. The combination of the microcontroller 110 
and the control logic program 115 constitutes a control 
module. As will be discussed, the control module may be 
implemented using hard-coded logic and other equivalent 
structures. The microcontroller 110 is also coupled to a timer 
module 120. In most embodiments the timer module 120 is 
implemented using hardware and software structures avail- 
able within the microcontroller 110 itself. The microcon- 
troller is also coupled to the standard computer 104. In the 
illustrative embodiment, three interface structures are 
shown. The microcontroller is coupled to a computer mouse 
interface module 125, a keyboard interface module 130, 
and/or a port interface module 135. In a given embodiment, 
one or more of these interface modules may be imple- 
mented. The interface modules 125, 130 and 135 are pref- 
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erably coupled to a user-ioput device. A user-input device is the monitor in the computer system 104. Other embodiments 

any device such as a mouse, a keyboard, a joystick, or a may be constructed where the proximity-sensor interface 

microphone which accepts inputs in response to actions of a system 102 is built into a computer display monitor, a mouse 

user. The user-input device is typically designed to connect or a trackball, or some other add-on accessory device. The 

to an associated computer-input port such as a mouse port, 5 present invention comprehends all of these variations as well 

a keyboard port, a joystick port, a microphone port, an as other equivalent variations which may be deemed desir- 

infrared port, or a wireless interface. able as new accessories and peripherals are added to com- 

The proximity sensor 105 may be implemented using puter systems, 
various component technologies. For example, the proxim- Various configurations may be employed to provide 
ity sensor 105 may be implemented using a passive infra-red 10 power to the proximity -sensor interface system 102. Often 
sensor, a diffuse reflectance sensor, a reflectance sensor, a the proximity sensor 105 will require one or more direct 
light beam continuity sensor, a capacitance sensor, a radio current (DC) voltage levels other than the ones readily 
frequency (RF) sensor, an audio sensor, an ultrasonic sensor, available. In such cases a DC-to-DC converter may be used 
a pressure sensitive mat, or a weight sensor within a chair. to transform an available voltage such as +5 volts to some 
Any sensor which can detect the physical presence of a user l5 other needed voltage. Other known methods are also avail- 
is within the scope of the present invention. able to derive a given voltage level from an alternating 

In various systems any of these sensors may be used in current (AC) power source, 
combination. A sensor made up of more than one sensor is The proximity-sensor interface system 102 controls a flow 
known as a multidimensional sensor since it provides a of information between the proximity sensor 105 and the 
multidimensional vector of data. A multidimensional sensor 20 standard computer 104, In the illustrative embodiment of 
made up of two or more proximity sensors is called a FIG. 1, the flow of information is controlled using the 
"multidimensional proximity sensor." For example, a mul- combination of the microcontroller 110 and the control 
tidimensional sensor may be constructed using a combina- program logic 115. In the illustrative embodiment, the 
tion of a reflectance sensor and a RF sensor. As will be control program logic 115 is implemented as a software 
discussed in connection with FIG. 5, the reflectance sensor 25 program held within a memory module (not shown). In some 
allows any individual to be detected and the RF sensor embodiments, the control program logic 115 may reside in 
allows the individual to be identified. Another example of a an internal memory located within the microcontroller 110. 
multidimensional sensor is a charge-coupled device (CCD) The control program logic 115 may also reside in a static 
image sensor as used in a video camera. As video confer- memory such as a read only memory (ROM) or an electri- 
encing protocols such as the H.323 protocol from the 30 cally erasable read only memory (EEPROM). In embodi- 
International Telecommunications Union (ITU) become ments involving an EEPROM, the control program logic 115 
increasingly prevalent, more and more computers such as may be optionally downloaded via one of the interface 
the standard computer 104 will include a CCD camera. In modules 125, 130 and/or 135 to allow modifications to be 
such cases, the existing CCD camera may be used as the made. In still other embodiments, the control program logic 
proximity sensor 105 if the control logic program 115 is also 35 115 may be downloaded across one of the interface modules 
employed. In a system using the existing CCD camera as the 125, 130, or 135 into a volatile memory such as an SRAM 
proximity sensor 105, an image processing software pro- "chip. The microcontroller 110 may be implemented using an 
gram would cooperate with the control logic program 115 to 87C751 from Phillips Semiconductor Inc. Other 
recognize the presence and possibly the identity of the user. microcontrollers, such as the PIC series of microcontrollers 
The proximity sensor 105 may also be implemented with 40 available from Microchip, Inc., may also be used to control 
similar technologies such as a microphone coupled to a the flow of information between the proximity sensor 105 
voice recognition software program. A microphone can be and the standard computer 104. This flow of information 
used to collect a vector of speech samples and thus behaves may similarly be controlled using any available micropro- 
like a multidimensional sensor. Voice recognition software cessor or microcontroller. 

analyzes a user's voice input and compares this input to a 45 The control module made up of the combination of the 

"voice print" to identify the user. Because most multimedia microcontroller 110 and the control logic program 115 may 

PCs already include microphones, solutions augmented with be alternatively implemented using one or more customized 

microphone based processing can be provided without the logic devices. For example a programmable logic array, a 

need for additional hardware. The embodiments making use gate array or an application specific integrated circuit 

of multidimensional sensors and signal processing tech- 50 (ASIC) may be designed using a hard-coded state machine 

niques are discussed in connection with FIG. 5. to control information flow and perform related logic pro- 

The proximity sensor 105 may be physically mounted cessing. Most hard-coded logic based solutions do not 

within the computer system 100 in various ways. In one require a separate memory to hold the control program logic 

embodiment, the proxim ity -sensor interface system 102 is 115 since that logic is hard-coded. The hard-coded logic 

built within its own enclosure as a stand-alone unit. Such an 55 based solution represents an engineering trade-off between 

embodiment is illustrated in FIG. 1A where the present design expenses and production expenses. When the pro- 

invention is embodied as a stand-alone device which con- duction volume warrants, it often becomes economical to 

nects between the mouse 182 and the computer system 104. pursue the hard -coded logic based solution. In this, case, the 

Other embodiments involve building the proximity-sensor microcontroller 110 and the control program logic 115 arc 

interface system 102 into a keyboard or a mouse. The 60 implemented using the hard -coded logic device. It should 

computer system 104 as illustrated in FIG. 1A includes a also be noted that some hard-coded logic devices are also 

keyboard, a monitor and a mouse 182. Still other embodi- reprogrammable. For example, the control program logic 

ments involve building the proximity-sensor interface sys- 115 may be downloaded via the interface module 125, 130 

tern 102 into the enclosure used to house a motherboard of or 135 into a set of volatile logic-configuration-memory 

the computer. This enclosure is commonly called a "CPU 65 cells located on a gate, array as provided by Xilinx Corp. 

box" or equivalently, a "CPU tower." In FIG. 1A, the "CPU Such solutions have the advantage of being easily modified 

box" is shown as being located between the keyboard and and upgraded. 
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The timer module 120 is preferably implemented in both implementations represent different price-to-performance 

hard -coded designs and microcontroller based designs using ratios and will be dictated by the needs of an individual 

an internal programmable timer module. An internal timer embodiment. The bus 140 also comprehends multi-layered 

includes a register to hold a count value, a counter state bus structures. For example, some embodiments make use of 

machine, and a coupling to a clock input. Such timer 5 a local processor bus connected to the CPU 145 and the 

modules are well known and are available with most micro- memory and storage device 150. A peripheral interconnect 

controllers. For example, the 87C751 microcontroller from bus such as a PCI bus is then used to interconnect the other 

Phillips Semiconductor Inc. has an internal programmable subsystems. In multi-layered bus based designs, the different 

timer module. This timer module is documented in the i aye rs are preferably interconnected by bus bridges. All of 

87C751 microcontroller user's guide available from Phillips 1Q these and other equivalent embodiments of the bus 140 are 

Semiconductor Inc. known to the skilled artisan and are considered to be 

Depending on the embodiment, one or all of the interface equivalents of the bus 140. From here forward, the discus- 
modules 125, 130 and 135 may be implemented. In an sion will center on the illustrated embodiment of the stan- 
exemplary embodiment, the present invention is imple- dard computer 104 whereby all subsystems are directly 
mented using only the mouse interface module 125. This 15 connected via the bus 140. Embodiments where the bus 140 
embodiment is discussed in detail in connection with FIG. represents a different physical interconnection topology are 
1A. Other embodiments use only the keyboard interface implicitly included in the discussion below, 
module 130. Various types of embodiments may be con- i D operation, the standard computer 104 operates a screen"^ 
structed using the port interface module 135. The port saver program 155. For example, whenever the user moves 
interface module may be implemented using a serial port 20 0 r clicks the mouse, information indicative of these actions 
such as an RS232 port, a parallel port such as a printer port, are transmitted via the mouse interface module 125 to the 
an infrared port, a wireless interface, or some specialized CPU 145. In general, the user provides an input via a user 
port such as microcontroller port interface. Similarly, a inputK)utput device and the user input-output device trans- 
universal serial bus interface (USB) as is used to intercon- m i ts the information in the form of a coded-command signal 
nect various types of computer peripherals may also be used. 25 ma de up of a sequence of voltage-levels representative of 
The port interface module 135 comprehends any custom or bits of the information. Typically, interrupts are generated to 
standardized port interface which may be used to route m f orm t he CPU 145 when the user's input has been pro- 
signals generated by the proximity-sensor interface system v ided. Likewise, whenever the user applies a keystroke to 
102 into the standard computer 104. tne keyboard, data indicative of this activity is transmitted to 

Referring again to FIG. 1, a set of component subsystems 30 the CPU 145 via the keyboard interface module 130. The 

within the standard computer 104 are illustrated as being ' screen saver program 155 is typically resident in the 

interconnected via a bus structure 140. For example, the memory and storage unit 150 and functions as a memory 

computer mouse interface module 125, the keyboard inter- resident background process within the software structure of 

face module 130, and the port interface module 135 are all the computer system 100. A timer is maintained and is 

internally coupled to the bus 140. Also coupled to the bus 35 updated in accordance with a timer-based interrupt. The - 

140 are a central processing unit (CPU) 145, a memory and timer is restarted with an initial count when a keyboard or \j< 

storage device 150, a screen saver program 1£5, a display , mouse stroke is detected. If no keyboard or mouse input is 

monitor 160, and an optional modem or network connection detected for the duration of a prespecified time-out period 

165. The screen saver program 155 preferably includes a the screen saver is moved from a background state into a 

screen saver activation control program and a screen saver 40 foreground state. When the screen saver functions in the 

display program. The screen saver activation control pro- foreground state, it displays a static or animated pattern on 

gram monitors user inputs and activates the screen saver the display monitor 160. The portion of the screen saver 

display program after a period of user inactivity has been which functions in the foreground state is also called a 

detected. A mouse is generally connected to the mouse "screen saver display program." The portion of the screen 

interface module 125 and a keyboard is generally connected 45 saver program with operates in the background state is also 

to the keyboard interface module 130. ca lled a "screen saver activation control program," The 

In alternative embodiments, the screen saver program 155 screen saver activation control program monitors user inputs 
maybe any user activity controlled computer program. Also, and places the screen saver program into the foreground 
more than one such program may be employed in a single state after a period of user inactivity has been detected, 
system. For example, the screen saver program 155 of the 50 Typically, even when the screen saver display program has 
computer system 104 may involve both a conventional been placed into the foreground state, the screen saver 
screen saver program and a separate password protection activation control program continues to run in the back- 
program. In other embodiments, the screen saver program ground. When a user input or physical presence is once again 
155 may include a password protection feature. The discus- detected the screen saver activation control program even- 
sion herein will focus on the illustrative embodiment as 55 tually places the screen saver display program back into a 
depicted in FIG. 1. background state. When the screen saver display program is 

The bus 140 may be constructed as a set of unbroken -in the background state it is substantially inactive. *" s ' 
wires or one or more optical fibers used to carry signals In embodiments of the present invention involving other 
between the component subsystems within the standard types of programs beside screen saver programs, the "screen 
computer 104. In some embodiments of the present 60 saver display program" may be replaced by a more generic 
invention, the bus 140 may be implemented equivalently "operational program." For example, the activation control 
using a set of direct parallel and/or serial connections program 230 may control the activation of a "password 
between individual modules. The bus 140 as illustrated in protected access program." In this case, the "operational 
FIG. 1 shows a low cost means to connect the illustrated program" is the "password protected access control pro- 
subsystems. A combination of bus connections and direct 65 gram." The "password protected access control program" is 
serial or parallel links may be used to implement a subset of a program which restricts access to one or more computer 
the connection structure provided by the bus 140. Different programs unless a password is properly entered. Similarly, 
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the screen saver display program may itself include a may be programmed to enter a "training mode." In one 

password protected access control program. embodiment of a training mode, a message is displayed 

In the above discussion, the notions of a "background asking the user to strike one or more keys to achieve a 
state" and a "foreground state" have been used. A specified effect such as deactivating a screen saver. When 
"background-processing module" is a computer program 5 the user strikes the one or more keys, the microcontroller 
which runs in the background state and is allowed to run 110 records associated keystroke information and subse- 
concurrently with another program. Usually, the operation of quently uses this information to emulate user activity, 
a background process is transparent to the user. A Alternatively, the user may activate a switch to enter the 
"foreground -processing module" is a computer program training mode. More details regarding the operation of 
which runs in the foreground state and whose operation is 10 specific embodiments of the proximity-sensor interface sys- 
evident to the user. A foreground-processing module is tem 102 are provided in connection with FIGS. 1A to 5. 
typically operative to present a user interface to the user. As While it is advantageous to be able to add the proximity- 
discussed above, the screen saver program 155 includes both sensor interface system 102 to the standard computer 104 
a background -processing module and a foreground- without the need to alter the design of the hardware or 
processing module and can thereby operate in both the 35 software of the standard computer 104, in some cases it may 
background state and the foreground state. While the user is be advantageous to do so. For example, when the present 
actively using the standard computer 104, the screen saver invention is provided as a standard feature of a computer, it 
program 155 does not actively display a screen saver image is advantageous to alter the hardware and/or software of the 
on the display 160. Rather, it quietly monitors the keyboard standard computer itself. In this case instead of supplying 
and the mouse interface activity. A counter is allowed to run 2 o the standard computer 104 and augmenting it with the 
and is reset every time a user input is detected. When no user proximity-sensor interface system 102, the computer system 
inputs have been detected for a long enough time to allow 100 is manufactured and supplied as an integrated system. In 
the counter's time-out value to be reached, the screen saver such a system the screen saver activation logic which runs 
program 155 moves into the foreground and makes its in the background is preferably implemented using the 
presence known by displaying the screen saver image or 2S methods of the present invention. In this situation the 
animation. When the screen saver moves from the back- proximity-sensor interface system 102 preferably makes use 
ground to the foreground, it is said to have been "activated." 0 f the port interface module 135, the bus 140 or other 
When the screen saver moves from the foreground to the internal interface. The proximity sensor then provides input 
background, it is said to have been "deactivated." to the interrupt structure of the operating system just like the 

In accordance with the present invention, the proximity- 30 keyboard and mouse. The background-processing module 

sensor interface system 102 augments the aforementioned within the screen saver program 155 then updates a set of 

operation by preventing the screen saver program 155 from variables needed to determine when to activate the screen 

being activated while the user remains in the vicinity of the saver program 155. The set of variables is included in a data 

computer system 100. When the proximity sensor 105 area which is preferably co-located along with the screen 

detects a user to be present, the control logic program 115 35 saver program 155. The logic used to control screen saver 

instructs the microcontroller 110 to emulate user activity by activation in such an embodiment is described in connection 

applying a data sequence to one of the interface modules with FIG. 4. Such a modified screen saver program can be 

125, 130, or 135. For example, in some systems the micro- built into an operating system as discussed in connection 

controller 110 interfaces to the standard computer 104 via with FIG. 2. Before discussing these embodiments, an 

the mouse interface module 125. When the proximity sensor 40 embodiment of the proximity-sensor interface system 102 

10 detects the user to be present, the microcontroller 110 which plugs into an existing mouse connector of the stan- 

signals the user's presence by supplying a mouse-data signal dard computer 104 is described. 

indicating to' move the cursor one or more pixels in the Referring now to FIG. 1A, a detailed schematic of a 
upward direction. A short time later, the microcontroller specific embodiment 190 of the proximity-sensor interface 
sends a mouse-data signal indicating to move the cursor 45 system 102 is illustrated. In this embodiment, only the 
back downward to the original position. The screen saver's mouse interface module 125 is used, and the microcontroller 
counter is thus prevented from reaching its time-out value 110, the control program logic 115 and the timer module 120 
and the screen saver is prevented from activating. This are all implemented using an 87C751 microcontroller 112. 
added functionality is advantageously added without the The microcontroller 112 is coupled to receive an input from 
need to alter the screen saver software 155 or the standard 50 the proximity sensor 105. An oscillator circuit 107 is also 
computer 104. provided and is coupled to generate a clock signal for the 
In some systems, a mouse data emulation of one pixel microcontroller 112. In this embodiment, the oscillator 107 
movement is sufficient to prevent the user activity control is made up of a crystal and two capacitors which are coupled 
from timing out. In other systems, depending on the screen in a feedback arrangement to an amplifier located within the 
saver activation software, an emulation of a movement of 55 microcontroller 112. This combination produces a square 
more than one pixel is required. In still other screen savers, wave oscillator which provides a clock signal to the micro- 
a one pixel movement will delay the screen saver activation, controller 112. In the illustrative embodiment, the mouse 
but is insufficient to "wake up" the computer from a screen interface module 125 is designed using the PS/2® mouse 
saver foreground state. In yet other systems, a keyboard interface protocol. An MTA41110 mouse and trackball con- 
command such as a "control" keystroke or a "function" 60 trailer IC from Microchip Inc. may be used to control this 
keystroke must be emulated. The microcontroller 110 may interface. The MTA41110 data sheet provides a detailed 
be programmed by skilled artisans to emulate the appropri- description of such an interface together with background 
ate signals needed to control the activation of screen savers information useful for implementing the mouse interface 
and other user activity controlled programs. For example, module 125 of the present invention. A first connector 170 
the microcontroller 110 may be preprogrammed during 65 is designed according to the PS/2® mouse protocol and is 
manufacture to generate the proper signals to control a connectable to a mouse-port as provided by the standard 
screen saver program. Alternatively, the microcontroller 110 computer 104. A second connector 175 is also designed 
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according to the PS/2® mouse protocol and is connectable 
to a PS/2® compatible mouse device 182. The mouse 
interface module 125 is thus designed to connect between 
the standard computer 104 and the mouse device 182. 

The internal wiring employed within the embodiment 190 5 
to interface the first connector 170 to the second connector 
175 is illustrated in FIG. 1A. A +5 v power wire, a ground 
wire, and a cable shielding wire are connected straight 
across the interface. The +5 volt and the ground wires are 
tapped off and routed to a voltage regulator 109. The voltage 10 
regulator 109 supplies a regulated +5 volt power lead and a 
ground lead to the microcontroller 112 as well as any other 
devices needing power. In some systems the proximity 
sensor 105 requires DC voltage levels beside the ones 
shown, in which case a DC-to-DC converter may be advan- j ^ 
tageously used. 

The data and clock wires do not pass straight across the 
interface but are rather routed to a set of input-output port 
pins of the 87C751 microcontroller 112. A first pair 180 of 
data and clock wires are connected between first connector 2 o 
170 and a first pair of input-output port pins of the micro- 
controller 112. A second pair 185 of data and clock wires are 
connected between the second connector 175 and a second 
pair of input-output port pins of the microcontroller 112. The 
proximity sensor 105 is also coupled to provide a digital 2 s 
logic input to another port pin of the microcontroller 112. 
The wires carrying the data and/or the clock signals carry the 
"mouse-data signal" used to indicate a mouse movement. 
Note that a keyboard interface is very similar to a mouse 
interface, and embodiment 190 may be modified for the case 30 
where the data and clock wires carry keyboard-data signals 
indicative of keystroke activity. 

In operation, the microcontroller 112 receives a set of data 
and clock signals from the mouse device 182 on the second 
pair 185 of data and clock wires. The microcontroller then 35 
echoes these received signals by transmitting them back to 
the standard computer 104 via the first pair 180 of data and 
clock wires. For example, when a user moves the mouse 
device 182 the data and clock wires 185 become active by 
carrying signal information which the microcontroller 112 40 
will detect on its input-output port pins. The microcontroller 
112 is operative to read the received data values into an 
internal register. Also, the microcontroller 112 echoes the 
data and clock signals back out the second data and clock 
wires 180. This way, the standard computer 104 receives 45 
inputs sent out by the mouse device 182. Since the micro- 
controller 112 passes the data signals generated by the 
mouse device 182 back to the standard computer 104, the 
microcontroller 112 is able to monitor mouse activity. Since 
the microcontroller 112 is able to drive the first pair 180 of 50 
data and clock wires, it is also able to generate its own data 
patterns and deliver them to the standard computer 104 via 
the first pair 180. 

As discussed in connection with FIG. 4, the microcon- 
troller 112 is operative to monitor data traveling from the 55 
second pair 185 of data and clock wires 185 back to the first 
pair 180 of data and clock wires. When a period of inactivity 
is detected, and the proximity sensor detects the physical 
presence of the user, the microcontroller 112 is operative to 
insert its own data sequence and deliver it via the first pair 60 
180 of data and lock wires back to the standard computer 
104. The data signal sent back to the standard computer 104 
typically signals a mouse movement of one or more pixels 
in the up and/or down direction. Other movement directions 
such as a left and/or right may also be signaled. When this 65 
data signal is received, the screen saver background- 
processing module within the screen saver program 155 
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resets its counter as though the user had moved the mouse 
device 182. In essence, the screen saver background- 
processing module is fooled into believing the user had 
moved the mouse when in fact the user was detected as 
being in the vicinity of the computer by the proximity sensor 
105, but no mouse movement was actually made. This 
allows the user to tend to other tasks such as the telephone 
without the screen saver being activated. In some embodi- 
ments the screen saver program 155 may include a password 
protection feature, and in other embodiments the screen 
saver program 155 may be substituted or augmented with 
other user activity controlled programs. More discussion 
relating to aspects of the operation of the proximity-sensor 
interface system 102 and the embodiment 190 are discussed 
in connection with FIG. 3-6. 

Referring now to FIG. 2, a software architecture 200 is 
illustrated which can be used to control the hardware of the 
computer system 100. The software architecture 200 is 
applicable to systems designed to accept inputs from the 
proximity-sensor interface system 102 via the port interface 
module 135. This input is supplied to the software architec- 
ture via an input 214. The software architecture 200 is most 
applicable for computers where the proximity-sensor inter- 
face system 102 is integrated into the system at the time it 
is manufactured as opposed to standard computers to which 
the proximity sensor system 102 has been added at a later 
time. The software architecture 200 may be supplied on a 
recordable medium readable by a computer system. An 
example of the recordable medium is a CD-ROM. The 
CD-ROM is inserted into a CD-ROM disc-drive, and the 
software with the software architecture 200 is loaded into 
volatile and/or non-volatile memory in the storage unit 150 
of the computer system 100. When the computer system 100 
is shipped with software loaded, the software with the 
software architecture 200 may be already loaded into the 
memory and storage unit 150. 

At the center of the software architecture 200 is an 
operating system kernel 205. Operating system kernels are 
well known in the art and control the access of software 
processes to the CPU 145. A software process can be defined 
as a flow of instructions executed on the CPU 145. As is 
common practice, the kernel 205 maintains a data structure 
called a task control block for each process. The kernel 
accepts interrupt inputs from a set of input-output sources 
210 and a timer 215. In the embodiment shown, the set of 
input-output sources 210 includes a keyboard 212 a mouse 
213 and a proximity sensor input 214. The kernel 205 
controls the execution of multiple programs on the CPU 145 
by activating and deactivating processes in response to the 
interrupt inputs produced by the set of input-output sources 
210 and the timer 215. An example of a process which is 
activated and deactivated as a function of interrupts is a 
screen saver display program 220. The activation and deac- 
tivation of a set of processes 235 corresponding to device 
drivers and user programs are also controlled using inputs 
based on the interrupts supplied by the set of input-output 
sources 210 and the timer 215. For example, in many 
operating systems, when a user moves a cursor into a 
window using a mouse, the program represented by the 
window is activated. An example of an available operating 
system kernel is the Unix® kernel originally developed by 
Bell Laboratories of AT&T, now Lucent Technologies, Inc. 
Other examples of operating system kernels are the kernels 
within the Windows98® and WindowsNT® operating sys- 
tems provided by Microsoft Inc. Hence the software archi- 
tecture 200 is preferably implemented as a computer oper- 
ating system such as one which is shipped with the computer 
system 100. 
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The software architecture 200 is operative to control the be incorporated into the method 300 to allow the user to 

activation of the screen saver display program 220 by taking leave the work area for a specified amount of time while 

into account input provided by the proximity-sensor inter- preventing an attached software system to time-out. For 

face system 102 as supplied by the port interface module example, the user can set a variable to indicate the method 

135. That is, a set of methods as discussed in connection 5 300 is to continue to prevent the on-line connection program 

with FIG. 3-5 may be practiced by an activation control from disconnecting for twenty minutes after the user has left 

program 230 which is a background-processing module and the vicinity of the computer. 

controls the activation of the screen saver display program FIG. 4 illustrates a method 400 which corresponds to a 
220. In the software architecture 200, an activation control specific embodiment of the method 300. The method 400 is 
program 230 is operative to analyze inputs from multiple 10 preferably implemented as a part of the control program 
sources to determine when to place the screen saver display logic 115. The method 400 preferably runs on the micro- 
program 220 into an activated foreground state whereby it controller 110 and exercises the timer module 120, The 
displays a screen saver. That is, the software architecture 200 method 400 can also be implemented as a control program 
involves an operating system and an activation control running on the microcontroller 112 in the specific embodi- 
program 230 which processes information provided via the 15 ment 190. The method 400 is operative to provide additional 
port interface module 135. The activation control program control to a screen saver program running on the standard 
230 preferably analyzes the keyboard input 212, the mouse computer 104. The method 400 augments presently avail- 
input 213 and the proximity sensor input 214 in determining able screen saver activation logic with inputs provided by 
when to activate the screen saver display program 220. the proximity sensor 105. When the user is detected by the 
As discussed in connection with FIG. 5, other user 2 o proximity sensor 105, the method 400 periodically transmits 
activity controlled programs beside the screen saver display information to emulate a user input. In the embodiment 190, 
program 220 may be similarly activated by a program with the transmitted information indicates small mouse 
the structure of the activation control program 230. For movements, for example one or more pixel upward and/or 
example a usage statistics monitoring system may be imple- downward periodically at a regular thirty -second interval, 
mented to keep statistics relating to a worker's presence in 2 s T° e interval is controlled by the timer module 120 which 
the work area, and a security system may be implemented to uses timer-data and a counting procedure to periodically 
detect and report intruders during times where no use of the supply a time-out pulse at a prespecified interval. The net 
computer system 100 is authorized. These other uses are effect is to keep the screen saver from being activated 
discussed in connection with FIG. 5. In other systems, the without distracting the user and without needing the hard- 
screen saver display program 220 may be substituted by or 30 ware or the software of the standard computer 104 to be 
augmented with a password protection program. In general, modified. 

the screen saver display program 220 may be replaced by in a first step 405, a set of user inputs are sampled or 

any operational program whose activation is controlled by otherwise recorded. The sampling of inputs represented by 

the activation control program 230, me first step 405 may be asynchronous. That is, the user 

FIG. 3 illustrates a method 300 used to control access to 35 inputs sampled in the first step 405 may be tied to interrupts 

a computer program such as a screen saver. In a first step and thus arrive outside of the control flow illustrated by the 

305, an input is checked based on the output of the proximity flowchart of the method 400. The method 400 is itself 

sensor 105. Control passes out of the first step 305 based preferably tied to a timer interrupt. In a preferred 

upon a decision 310. If the proximity sensor does not detect embodiment, the step 405 represents the entry point to an 

a person to be present, no action is taken and control loops 40 interrupt service routine triggered by a time-out signal 

back to the first step 305. If the proximity sensor does detect generated by the timer module 120. The timer module 120 

a person to be present, control passes to a second step 315. is preferably configured to generate the time-out signal once 

In the step 315, an action is taken to prevent a program such every second. This way, the step 405 is executed once per 

as a screen saver from being activated. As discussed in second. Control passes from the first step 405 based on a first 

connection with FIG. 1A, different embodiments can be 45 decision 407. Tlie first decision 407 is operative to pass 

used to prevent the screen saver from being activated by control back to the step 405 if the proximity sensor 105 

manipulating data sequences passed across an interface such detects the user not to be present in the vicinity of the 

as the mouse interface 170. As discussed in connection with computer 100. In the preferred embodiment, the timer 

FIG. 2, a screen saver can be prevented from being activated module 120's interrupt introduces a one second delay into 

by analyzing the proximity sensor inputs 214 in addition to 50 the control path from the first decision 407 back to the first 

the keyboard inputs 212 and the mouse inputs 213. The step 405. When this control path is taken, no inputs are 

method 300 may similarly be applied to control the activa- provided by the method 400 and the screen saver activation 

tion of other programs beside a screen saver. program is allowed to function in its normal mode. 

Any program whose activation is normally governed by If the proximity sensor 105 does detect the user to be 

the detection of the absence of a keyboard or mouse input 55 present, control passes from the first step 405 based on a 

from a user may be modified in accordance with the method second decision 408. The second decision 408 evaluates to 

300. An example is an Internet browser or other on-line true if a user input has been detected. For example, in the 

connection program. If the user does not provide any input embodiment 190, the second decision 408 evaluates to true 

for a time, the on-line connection program will typically if a mouse movement has been detected. In other 

disconnect the user. This causes the user to need to recon- 60 embodiments, the second decision 408 is based upon inputs 

nect. Reconnecting can often be a tedious and annoying task, from one or more other sources such as a keyboard. When 

especially when the on-line system becomes congested and the second decision 408 is affirmative, control next passes to 

denies the user access. The method 300 can be directly a third step 410. In the third step 410, a counter variable is 

applied to this type of application by detecting when the user reset. In the embodiment 190, this counter variable is 

is present in the work area and causing data to be transmitted 65 included in a data area controlled by the method 400 and is 

over a communication link indicative of a keystroke or other preferably set to a value of thirty. Control next passes from 

input. As discussed in connection with FIG. 4, a timer may the second step 410 to a third step 415. If no user input is 
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detected in the decision 408, control passes directly from the 
first step 405 directly to the third step 415. In the third step 
415, the counter variable is decremented. Note when user 
input is detected in the decision 408, the counter is reset in 
the step 410 so as to inhibit the counter from decrementing 5 
to zero when a user is actively supplying inputs to the 
computer system 100. 

Control next passes from the third step 415 based on a 
third decision 417. In the third decision 417, the counter 
variable is compared to zero. If the timer variable has not 1Q 
decremented to zero, control passes from the third decision 
417 back to the first step 405. In the preferred embodiment, 
this branch incurs a one-second delay because the reentry of 
control into the step 405 is controlled to coincide with the 
timer-out signal produced by the timer module 120. If the 
counter variable is detected as having counted down all the 15 
way to zero, control passes from the third step 415 to a 
fourth step 420. The fourth step 420 is operative to send a 
signal to simulate a user input. In the illustrative embodi- 
ment 190, the fourth step 420 is operative to send a simu- 
lated mouse command over the first pair 180 of data and 20 
clock wires to emulate a mouse movement. An internal state 
variable is also preferably toggled in the step 420 so the 
method 400 can keep track of whether to send a command 
to move the mouse upward and/or downward by one or more 
pixels. After the fourth step 420 has executed, control next 2 5 
passes back to the first step 405, again with a one-second 
delay. 

The effect of the method 400 is to send mouse movement 
commands and to prevent the screen saver from activating 
when the user is detected as being present in the vicinity of 30 
the proximity sensor 105. In a preferred embodiment as 
described above, the method 400 sends a mouse-data com- 
mand at a regular interval of 30 seconds to toggle the cursor 
position upward or downward by one or more pixels. In 
other preferred embodiments the cursor position may be 35 
toggled in other ways, such as, for example left and/or right 
by one or more pixels. When the user is actually providing 
user inputs, the counter variable is reset in the step 410 
thereby inhibiting the transmission of additional user input 
signals in the step 420. Only when the user has not provided 40 
any input for thirty seconds will a mouse movement be 
emulated in the step 420. 

While the method 400 is designed to operate with the 
screen saver activation program provided by the standard 
computer 104, the method 400 also may be applied within 45 
the activation control program 230. This alternate embodi- 
ment corresponds to the case where the computer system 
100 is designed for use with the proximity -sensor interface 
system 102. Typically, such embodiments involve the port 
interface module 135, the bus 140, or other internal inter- 50 
face. The activation control program 230 accepts the inputs 
provided by the input-output sources 210. In an embodiment 
whereby the method 400 is implemented within the activa- 
tion control program 230, the first decision 407 is deleted 
and the second decision 408 checks to see if any activity has 55 
been detected on the input-output sources 210, If the input- 
output sources 210 report any activity, then control passes 
from the first step 405 to the second step 410. If no inputs 
are detected, control passes from the first step 405 to the 
third step 415. This way the method 400 can be applied to 60 
screen savers and operating systems specially designed to 
accept control from the proximity-sensor input 214. 
Moreover, while the method 400 is illustrated as controlling 
a screen saver display program, the method 400 may also be 
used to control any user activity controlled program. 65 

A variation of the method 400, a method 400A is illus- 
trated in FIG. 4A. The method 400A augments the method 



145 Bl 

16 

400 with the ability to automatically deactivate a user 
activity controlled program such as the screen saver program 
155 when the user returns to the computer system 100. For 
example, the user may immediately wish to check an email 
application for new messages. The method 400A is most 
applicable when the screen saver 155 does not require the 
user to enter a password. As discussed in connection with 
FIG. 5., the method 400Acan be also be modified to operate 
when the screen saver program 155 involves a password. 

The method 400A, begins with the first step 405. As with 
the method 400, this step is preferably entered once per 
second in coincidence with timer module 120' s time-out 
signal. In the method 400A, control passes from the step 405 
under the control of the decision 407. If the proximity sensor 
105 does not detect the user to be present, the decision 407 
regulates control to pass from the first step 405 to a fifth step 
430. The fifth step 430 is operative to increment a not- 
present counter. The not-present counter is preferably imple- 
mented as a software variable accessible by the control logic 
program 115 which implements the method 400A. The 
not-present counter keeps track of the amount of time the 
user has been away from the computer 100. If the user is 
gone a long time, the not-present counter preferably satu- 
rates at a maximum value to keep the counter from wrapping 
around to zero. In some embodiments the maximum value of 
the not-present counter may be as low as one. 

If the first decision 407 detects a user to be present, a 
fourth decision 431 checks the not-present counter value to 
a threshold. The threshold is a number greater than or equal 
to one which indicates an amount of time the user has been 
away from the computer system 100. If the not-present 
counter is greater than or equal to the threshold, then the 
decision 431 is operative to reset the not-present counter to 
zero and pass control from the step 405 directly to the step 
420. In the step 420 a pixel value is toggled as discussed in 
connection with the method 400, thereby immediately deac- 
tivating the screen saver program. If the not-present counter 
is below the threshold, control is regulated by the decision 
408 and the method 400A proceeds identically to the method 
400 as discussed above. 

The effect of the foregoing sequential logic is to provide 
the same functionality as the method 400, but to also 
deactivate the screen saver once the user has returned to the 
computer system 100. The threshold value is used to filter 
spurious events and may be set as low as one. Setting the 
threshold to one is equivalent to using a single-bit state 
variable to indicate the user was not present in a previous 
pass through the method 400A but is present in the current 
pass through the method. When the method 400A or an 
equivalent thereof causes an input to be sent to the screen 
saver program 155 (or the activation control program 230) 
substantially when the proximity sensor 105 detects a tran- 
sition from the user not being present to the user being 
present. 

Referring now to FIG. 5, a method 500 is illustrated for 
using the input provided by the proximity-sensor interface 
system 102 for additional modes of control. In a first step 
505 a set of program variables are set up and initialized. 
Control next passes to a second step 510. All control paths 
into the second step 510 are preferably regulated to coincide 
with the time-out signal produced by the timer module 120. 
For example, the second step 510 entered in response to a 
time-out interrupt produced by the timer module 120. In the 
second step 510 inputs provided by the proximity sensor 105 
are sampled. These inputs may be obtained by polling the 
proximity-sensor interface system 102, or by checking a 
memory location written under the control of an interrupt 
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handler. If no input is reported by the proximity sensor 105, present, the decision 512 evaluates to true and control passes 
control loops back around to the second step 510 under the to the third step 515. In the third step 515, the first radio- 
control of a decision 512. This looping of control preferably frequency transceiver is used to authenticate the identity of 
incurs a delay substantially equal to the time-out period of the detected person. For example, a security guard or an 
the timer module 120. If an input is reported by the prox- 5 authorized user may wear a security badge with a second 
imity sensor 105, control passes to an optional third step radio-frequency transceiver. The third step 515 is then 
515. The optional third step 515 is operative to perform a operative to cause an encrypted message to be transmitted 
user identification process as discussed below. Control next from the first radio-frequency transceiver. The second radio 
passes from the optional third step 515 to a fourth step 520. frequency transceiver located in the security badge then 
If an input is reported by the proximity sensor 105 and the 10 deciphers the message and produces an encrypted response, 
optional third step 515 is not present, control passes directly In one embodiment a plurality of encrypted messages are 
from the second step 510 to the fourth step 520. The fourth transmitted using a set of public keys for a set of authorized 
step 520 is operative to log statistics or take other actions. users. The third step 515 next sets a user-identification 
Examples of statistics logged or actions taken in the step 520 variable to indicate whether the detected person has been 
are presented below. 15 properly authenticated as an authorized user. In some sys- 

In a specific embodiment of the method 500, the fourth terns the user- identification variable also indicates the spe- 

step 520 is operative to act as a usage statistics monitoring cific identity of the detected person. Control next passes 

program which maintains a log of user activity within the from the third step 515 to the fourth step 520. In the fourth 

work area. In a working environment, a management entity step 520 an action is taken based on the user-identification 

may desire to measure worker productivity based on the 2 o variable. If the detected person did not pass the authentica- 

amount of time a worker is physically present within the tion process of the step 515, an alarm may be sounded and 

proximity of the computer system 100. In another specific - a message may be transmitted via the modem or network 

embodiment of the method 500, the fourth step 520 may be interface 165. If the detected person was identified as a 

set up to provide an intruder alarm. For example, the worker security guard, no further action is taken. If the detected 

may set an alarm-variable indicating that no user is autho- 2 s person is identified to be the authorized user of the computer 

rized to come in the work area for a specified amount of system 100 other actions may optionally be taken as 

time. Alternatively, the alarm-variable may be set based on described below. If a usage monitoring system is employed, 

a timer to indicate when no user is authorized to be in the the identity of the user may be taken into account while 

work area, or could also change the required password as a gathering statistics. For example, if system 100 is shared by 

function of time/day etc. When the proximity-sensor inter- 30 various users, the usage statistics monitoring program may 

face system 102 reports a person to be present, and the keep track of productivity of individual users, 

alarm-variable is set, an alarm may be sounded and a This aforementioned authentication system may be also 

message transmitted via the network or modem connection be preferably used to allow a user to reenter a work area 

165. Hence the method 500 provides a means for the present without the need to reenter a password into a screen saver, 

invention to add features to the basic screen saver activation 35 In one example system, the security policy of an institution 

control as described above. requires users log out of their computers at night, but allows 

Enhanced versions of the present invention are obtained screen savers with password to be used during the day. The 

when the optional third step 515 of the method 500 is method the method 400A is thus modified by adding authen- 

employed. Hie third step 515 is most useful when used with tication logic to the decision 431. If the not -present counter 

enhanced proximity sensors. As discussed in connection 40 indicates the user was away for at least the threshold amount 

with FIG. 1, the proximity sensor 105 may involve a of time, and the user has been properly authenticated as the 

plurality of different types of sensors arranged in a parallel authorized user, then a control variable is set. When control 

configuration. For example, the proximity sensor 105 may next passes to the step 420, the step 420 analyzes the control 

be constructed using both a passive infra-red sensor and variable, and if it is set, deactivates the screen saver. In 

radio frequency identification transceiver. Such a sensor 45 essence, the password is supplied by the second radio 

employs a plurality of different proximity sensor technolo- frequency transceiver located in the user's badge. If proper 

gies to provide a multidimensional output. A multidimen- challenge and reply protocols are used, an eves dropper 

sional output involves two or more output signals produced cannot intercept a transmitted password. This is because a 

by two or more sensors. In a multidimensional sensor, the different password is needed as a reply based on the 

individual sensors may be of the same or different types. An 50 encrypted challenge transmitted by the first radio frequency 

example of a multidimensional sensor using substantially transceiver. This way the only way for an unauthorized user 

identical sensors is an image sensor such as a charge coupled to be able to deactivate the screen saver is to get physical 

device (CCD) camera. A CCD camera provides a set of pixel possession of the user's badge. This is equivalent to the user 

values. Together the set pixel values constitute a video losing control of a physical key and deemed to be an 

image. The set of pixel values may be processed to deter- 55 acceptable risk in most environments. The automatic screen 

mine the characteristics of a scene as viewed by the CDD saver deactivation feature may also be automatically dis- 

camera. In many cases, the standard computer 104 may abled during certain hours of the day using computer date/ 

come equipped with a CCD camera to support video con- time clocking and a set of parameters to configure the system 

ferencing and related applications. 100 to implement a given security policy. 

In a preferred embodiment of an enhanced system, the 60 Referring now to FIG. 6, a method 600 is illustrated to 

method 500 is practiced using the proximity sensor 105 preprocess the outputs of low cost sensors to provide a 

which is implemented as a multidimensional proximity reliable "person-present" indication. While the method 500 

sensor and is constructed using both an infra-red sensor and and the use of multidimensional sensors support high-end 

a first radio-frequency transceiver. In this embodiment, the systems, there is also a need to support very low cost 

second step 510 is operative to check the infra-red sensor to 65 implementations. As such, the method 600 addresses the 

determine if a person in present in the vicinity of the problem of using a low cost version of the proximity sensor 

computer 100. If the infra-red sensor detects the person to be 105 which can only detect a delta, i.e., a change in envi- . 
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ronment. For example, a reflective proximity detector (RPD) 
may be employed to provide a highly reliable proximity 
detection, but such sensors are more costly than passive 
infra-red sensors. An RPD sensor includes a source of 
illumination such as an infra-red transmitter and then mea- 5 
sures the reflected signal level. A passive infra-red sensor is 
less expensive but is only reliable at detecting a delta such 
as when a change in heat occurs due to a person entering or 
leaving the area surrounding the computer system 100. A 
passive infrared sensor may be fooled especially in warm 10 
environments as the user moves about within the vicinity of 
the computer system 100. Thus without suitable pre- 
processing, the passive infrared sensor may not be able to 
provide a reliable "person present" indication as used in the 
decisions 310, 407 and 512. The method 600 is designed to 15 
process an input provided by a passive infra-red sensor so as 
to provide a reliable and low-cost "person-present" indica- 
tion. 

In a first step 605, a set of user values is sampled. The step 
605 is preferably entered based on a timed interrupt such as 20 
a one-half second interrupt. In such an embodiment, the step 
605 is entered every one-half second in response to a 
time-out signal. In some embodiments the method 600 may 
be incorporated into the logic of the step 305, 405 or 510. In 
such embodiments, a pass through the method 600 is 25 
allowed to run to completion before passing control from 
any of the steps 305, 405 or 510. The step 605 includes the 
sampling of the output of the proximity sensor 105. In some 
embodiments, the sampled values are passed through a filter 
such as a first order infinte-impulse response filter or a finite 30 
impulse response filter designed with a low-frequency pass- 
band using well known digital filtering techniques. Filtering 
differential values with a low pass filter tends to smooth the 
set of differential input-values to provide a set non- 
differential values. Control passes from the step 605 based 35 
on a decision 607. The decision 607 is affirmative if a mouse 
input has recently been detected. For example, the counter 
variable maintained in the steps 410 and 415 of the method 
400 may be evaluated in the decision 607. If the mouse has 
recently been used, then control passes to a step 610 which 40 
sets the person-present variable to true. This step ensures the 
person-present variable will be properly set at power-up 
time when the user first boots the machine. Preferably the 
counter used in the steps 410 and 415 is initialized to a large 
value at power-up time. Also, when a user does provide an 45 
input to the computer system 100, the second step 610 
assures the method 600 is reset into a correct and known 
state whereby the user is deemed to be present. Control next 
passes from the step 610 back to the step 605 through a 
delayed interrupt path. If the decision 607 detects that the no 50 
mouse input has recently been supplied, a second decision 
612 is next evaluated. In the decision 612, the proximity 
sensor is checked to see if a change, i.e., a delta, has been 
detected which is above a threshold. A threshold is used to 
filter out false alarms caused by the user moving around 55 
within the work area. When the delta has been detected, 
control passes to a step 615. The step 615 is operative to 
toggle the state of the person-present variable. That is, when 
a large delta is detected, the state is toggled. For example, if 
the user has been involved in using the computer system 60 
100, the person present variable will be properly set upon 
power-up. When the person becomes inactive and leaves the 
work-area, the person-present variable will change state to 
indicate no person is present. When the person reenters the 
work area, the person-present variable will toggle to indicate 65 
a person is once again present. In the event the system is 
fooled due to adverse conditions such as warm temperatures 
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which mask large deltas, the system will be restored to a 
correct state whenever a keystroke or mouse input is 
detected due to the logic leading into the state 610. Hence 
such preprocessing allows a low-cost sensor such as a 
passive infrared sensor to be used to provide a reliable 
screen saver activation system. 

In an alternative embodiment, the decision 607 checks to 
see whether a type of input other than a mouse input has 
been detected. For example, in some embodiments, the 
decision 607 is affirmative when a key on a keyboard has 
been recently used. 

Although the present invention has been described with 
reference to a specific embodiment, other embodiments may 
occur to those skilled in the art without deviating from the 
intended scope. While many of the illustrative examples 
discuss embodiments using screen saver activation pro- 
grams and screen saver display programs, the present inven- 
tion applies to systems involving any activation program 
which controls the activation and/or deactivation of any 
computer program. Also, the method 400 does not need to be 
tied to a one-second interrupt but an interrupt based on any 
time-out period may be used. Also, the fourth step 420 of 
method 400 could cause a pixel movement other than the 
one-pixel up/down toggle. The multidimensional proximity 
sensor can also be configured using various combinations 
beside the infra-red sensor and the radio-frequency trans- 
ceiver of the foregoing example. For example, a reflective 
proximity sensor could be used instead of the infra-red 
sensor. Lower cost radio frequency based sensors could be 
used. For example a simple low-cost transmitter could be 
located on the badge which transmits a given security code 
once per second and is inactive the rest of the time. Also, 
image and/or voice sensors may also be used. For example, 
if an image sensor is used, a face recognition module may 
be used to perform the user identification of the step 515 or 
the decision 431. Likewise a voice recognition speaker 
identification system may be used to allow a user to speak 
a password and have the password compared to the user's 
unique voice print. Also additional timer and filtering logic 
may be added to the method 600 to further enhance the 
ability of the passive infra-red sensor to reliably perform in 
this application. Therefore, it is to be understood that the 
invention herein encompasses all such embodiments which 
do not depart from the spirit and scope of the invention as 
defined in the appended claims. 

What is claimed is: 

1. Apparatus comprising: 

an interface module operative ly couple able to provide 

input to a computer; 
a proximity sensor operatively coupled to said interface, 

said proximity sensor operative to detect the presence 

of a user; and 

a control module, said control module controllably 
coupled to said proximity sensor and said interface 
module, said control module having a state machine 
operative to cause information emulating a user input 
from a peripheral device of the computer to be trans- 
mitted to said computer via said interface module based 
on an output signal provided by said proximity sensor 
to prevent the activation of a computer program. 

2. The apparatus according to claim 1, wherein said 
computer program is a screen saver program. 

3. The apparatus according to claim 1, wherein said 
computer program is a password protection program. 

4. The apparatus according to claim 1, wherein said 
computer program is an on-line connection program. 
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5. The apparatus according to claim 1, further comprising 
a timer, said timer being coupled to said control module, 
wherein said control module is operative to regulate when 
said information is transmitted based on data provided by 
said timer. 5 

6. The apparatus according to claim 5, wherein said 
regulation causes said information to be transmitted at 
regular intervals when said proximity sensor detects said 
user to be present. 

7. The apparatus according to claim 6, further comprising: no 
a coupling interposed between said control module and a 

user input-output device, said coupling operative to 
carry a signal indicative of the presence of an input 
produced by said user input-output device; 
whereby said regulation further inhibits said information 15 
from being transmitted at regular intervals when said 
proximity sensor detects said user to be present and 
said control module detects said signal. 

8. The apparatus according to claim 7, wherein said 
information comprises a mouse-data signal. 20 

9. The apparatus according to claim 7, wherein said 
information comprises a keyboard-data signal. 

10. The apparatus according to claim 1, wherein said 
control module is further operative to cause said information 

to be transmitted substantially when said proximity, sensor 25 
detects a transition from said user not being present to said 
user being present. 

11. The apparatus according to claim 1, wherein said 
output signal of said proximity sensor indicates a delta, and 
said control module is further operative to preprocess said 30 
output signal using a state machine to convert said delta into 

a user-present signal. 

12. Apparatus comprising: 

an interface module interpretable between a user-input 35 
device and a computer-input port of a computer, said 
interface module including a first connector connect- 
able to said user-input device and a second connector 
connectable to said computer-input port; 
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a proximity sensor operatively coupled to said interface 
module, said proximity sensor operative to detect the 
presence of a user; and 

a control module, said control module controllably 
coupled to said proximity sensor and said interface 
module, said control module having a state machine 
operative to cause information emulating a user input 
from a peripheral device of the computer to be trans- 
mitted to said computer via said interface module based 
on an output signal provided by said proximity sensor 
to prevent the activation of a computer program. 

13. The apparatus according to claim 12, wherein said 
computer program is a screen saver program. 

14. The apparatus according to claim 12, wherein said 
computer program is a password protection program. 

15. The apparatus according to claim 12, wherein said 
computer program is an on-line connection program. 

16. The apparatus according to claim 12, wherein said 
user-input device is a computer-mouse device, said input 
port is a computer-mouse port, and said information com- 
prises a data signal representative of a mouse movement. 

17. The apparatus according to claim 12, wherein said 
user-input device is a computer-keyboard device, said input 
port is a computer-keyboard port, and said information 
comprises a data signal representative of a keystroke. 

18. The apparatus according to claim 12, wherein said 
control module is further operative to pass a signal received 
from said first connector to said second connector and to 
monitor said signal to detect a user input. 

19. The apparatus according to claim 18, wherein said 
control module further comprises a timer, and said control 
module is further operative to cause said information to be 
transmitted at regular intervals when said proximity sensor 
detects said user to be present. 

20. The apparatus according to claim 19, wherein said 
control module is further operative to inhibit said informa- 
tion from being transmitted for a prespecified amount of 
time after said use input has been detected. 

* * * * * 
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